SAYLOR The result you are building
Finished Result:
A webhook receiver that authenticates the exact raw request, blocks replay, acknowledges within provider deadlines, durably queues work, processes idempotently, retries with bounded policy, and supports replay/reconciliation without duplicate business effects.
Use this guide when
- Receiving events from payment, auth, commerce, messaging, or other providers.
- Fixing missed, duplicated, forged, or slow webhook processing.
Do not use it as a substitute for
- Do not verify a reserialized body when the provider signs raw bytes.
- Do not perform slow/non-idempotent business work before acknowledging.
Before you change anything
Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Provider signature algorithm/headers/secret rotation and delivery docs.
- Raw-body framework behavior and endpoint exposure.
- Unique event/business IDs, ordering, retry window, and replay controls.
- Durable queue/storage, worker, reconciliation, and monitoring.
Stop Before Proceeding:
Stop production processing if signatures are not verified against current/overlap secrets and timestamp/replay rules, or if duplicate delivery can duplicate money/messages/orders.
Understand the system before fixing it
Delivery is generally at least once Duplicates and out-of-order events are normal conditions, not edge anomalies.
Authenticate before parsing into trusted fields Verify raw bytes, signature, timestamp, endpoint secret, and allowed origin/provider semantics before business use.
Acknowledge receipt, not completion Persist/queue durably then return success quickly; worker owns retries and effects.
Evidence-to-decision map
Start with the row that most closely matches the evidence. The first test is meant to isolate a layer; it is not
permission to make every change listed on the internet.
Evidence Likely layer First decisive check What the result means
Signatures always fail Raw Compare exact bytes, Framework parser or wrong environment
body/secret encoding, secret, timestamp secret changed payload.
Provider retries Acknowledgme Measure endpoint Response too slow/non-2xx or connection
despite processing nt status/latency failure.
Duplicate business Idempotency Replay same event/business Dedup store/atomic side-effect boundary
effect key concurrently missing.
State regresses Ordering Compare event created/version Older event overwrote newer state; fetch
and current object current source/reconcile.Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
01 Capture raw request safely Why: Signature verification depends on exact bytes.
Do: Configure endpoint to retain raw body and selected signature/timestamp headers with size/time limits; do not log payload/secrets by default.
Read the result: Raw bytes must be available before JSON parser mutation.
Next: Reject oversize/malformed transport.
02 Verify authenticity and replay window Why: A valid-looking JSON body is attacker-controlled.
Do: Compute provider-documented signature over exact bytes, constant-time compare against active/overlap secrets, validate timestamp tolerance and expected endpoint/environment.
Read the result: Failure returns no business processing.
Next: Support controlled secret rotation.
03 Persist and acknowledge Why: Provider timeout causes redelivery.
Do: Within deadline atomically store event ID/hash/status/raw reference per retention and enqueue job, then return documented 2xx.
Read the result: Duplicate same ID/hash returns accepted; ID/hash conflict alerts.
Next: Do not call slow downstream service inline.
04 Process idempotently Why: Queue and worker can redeliver after crash.
Do: Claim event atomically, map to business key/version, apply transaction/outbox/provider idempotency, store terminal result and attempts.
Read the result: One logical effect despite concurrent duplicates.
Next: Treat terminal vs retryable errors distinctly.
05 Handle ordering and source of truth Why: Delivery order may differ from creation order.
Do: Use event version/time cautiously; for mutable resources fetch current authoritative state when provider recommends; ignore stale transitions under state machine.
Read the result: Never let old event roll back confirmed newer state.
Next: Record decision.
06 Replay, reconcile, and observe Why: Missed/failed events need safe recovery.
Do: Provide authenticated replay by event ID, dead-letter review, provider reconciliation sweep, alerts for signature/latency/failure/age/duplicates, and load tests.
Read the result: Replay cannot duplicate effect and every event reaches terminal disposition.
Next: Document retention.
Worked example
Starting Problem:
Payment provider sends same success event three times after endpoint timeout.
Evidence collected
- Signature is valid.
- Handler charges fulfillment inline and takes 12 seconds.
- Provider deadline is shorter, so it retries.
- No durable event-ID uniqueness exists.
Decision Slow synchronous handling and missing idempotency cause duplicate fulfillment.
Actions taken
- Verified/stored/enqueued event then returned 2xx quickly.
- Added unique event ID and business payment ID with atomic fulfillment record.
- Replayed concurrent duplicates and simulated worker crash.
Proof Of Completion:
Provider receives prompt 2xx; three deliveries produce one fulfillment; crash/replay returns stored terminal result.
Why this example matters Reliability comes from accepting duplicates safely, not assuming they will stop.
Verify, recover, and hand off
Completion tests A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Raw-body signature, timestamp, environment, and rotation tests pass.
- Endpoint p95 acknowledgment fits provider deadline.
- Concurrent duplicate/replay creates one business effect.
- Out-of-order events cannot regress state.
- Retry/dead-letter/reconciliation reach terminal disposition.
- Secrets/payloads obey logging/retention policy.
Rollback or safe recovery
- Pause worker while still safely storing authenticated events.
- Return to prior handler/secret overlap configuration.
- Replay backlog only after idempotency/reconciliation is proven.
If the expected result does not appear What happened What it usually means Next safe move
Valid events rejected after deploy Body parser/order or secret Compare raw-byte middleware and active environment changed. secret.
Queue grows Worker/dependency slower than Scale/bound, apply backpressure, inspect arrival. oldest age/failure class.
Same ID different body Provider anomaly/attack/storage Reject/alert; never overwrite original. corruption.
Replayed old event changes state Ordering guard absent. Use version/state transition or fetch current source.
Reusable handoff record Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Provider contract, endpoint/secret environment.
- Raw verification and replay policy.
- Durable event/idempotency state model.
- Worker retry/ordering/reconciliation behavior.
- Load/failure/replay metrics and runbook.
Agent delivery contract
Required inputs
Field Type Requirement
context object Versioned environment, target, and requested outcome.
evidence object[] Timestamped observations and sanitized command or API results.
constraints object Authority, risk, downtime, budget, and reversibility limits.
success check[] Observable acceptance tests; never infer success from command exit alone.
Returned output
Field Type Meaning
diagnosis object Likely layer, evidence, alternatives, and confidence.
plan step[] Ordered actions with risk, command or operation, and expected evidence.
verification check[] Pass/fail checks that prove the requested outcome.
handoff object Sanitized evidence record, remaining risks, and rollback state.
Agent refusal and escalation rules
• Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
• Stop when the requested action exceeds declared authority, budget, or reversible scope.
• Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule
Score confidence from the number and quality of independent observations, not from how familiar the
error looks. Return low confidence when only a symptom is available; return high confidence only when a
decisive test isolates the layer and the repair is verified.Official reference starting points
- https://www.rfc-editor.org/rfc/rfc2104
- https://www.rfc-editor.org/rfc/rfc9110