SAYLOR Build webhook receivers that authenticate events, preserve raw bodies, prevent replay, acknowledge quickly, retry safely, and process idempotently.
The result you're building
A webhook receiver that authenticates the exact raw request, blocks replay, acknowledges within provider deadlines, durably queues work, processes idempotently, retries with bounded policy, and supports replay/reconciliation without duplicate business effects.
Use this guide when
- Receiving events from payment, auth, commerce, messaging, or other providers.
- Fixing missed, duplicated, forged, or slow webhook processing.
Do not use it as a substitute for
- Do not verify a reserialized body when the provider signs raw bytes.
- Do not perform slow/non-idempotent business work before acknowledging.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Provider signature algorithm/headers/secret rotation and delivery docs.
- Raw-body framework behavior and endpoint exposure.
- Unique event/business IDs, ordering, retry window, and replay controls.
- Durable queue/storage, worker, reconciliation, and monitoring.
Understand the system before fixing it
Delivery is generally at least once
Duplicates and out-of-order events are normal conditions, not edge anomalies.
Authenticate before parsing into trusted fields
Verify raw bytes, signature, timestamp, endpoint secret, and allowed origin/provider semantics before business use.
Acknowledge receipt, not completion
Persist/queue durably then return success quickly; worker owns retries and effects.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Signatures always fail | Raw body/secret | Compare exact bytes, encoding, secret, timestamp | Framework parser or wrong environment secret changed payload. |
| Provider retries despite processing | Acknowledgment | Measure endpoint status/latency | Response too slow/non-2xx or connection failure. |
| Duplicate business effect | Idempotency | Replay same event/business key concurrently | Dedup store/atomic side-effect boundary missing. |
| State regresses | Ordering | Compare event created/version and current object | Older event overwrote newer state; fetch current source/reconcile. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Capture raw request safely
Why: Signature verification depends on exact bytes.
Do: Configure endpoint to retain raw body and selected signature/timestamp headers with size/time limits; do not log payload/secrets by default.
Read the result: Raw bytes must be available before JSON parser mutation.
Next: Reject oversize/malformed transport.
Step 02 — Verify authenticity and replay window
Why: A valid-looking JSON body is attacker-controlled.
Do: Compute provider-documented signature over exact bytes, constant-time compare against active/overlap secrets, validate timestamp tolerance and expected endpoint/environment.
Read the result: Failure returns no business processing.
Next: Support controlled secret rotation.
Step 03 — Persist and acknowledge
Why: Provider timeout causes redelivery.
Do: Within deadline atomically store event ID/hash/status/raw reference per retention and enqueue job, then return documented 2xx.
Read the result: Duplicate same ID/hash returns accepted; ID/hash conflict alerts.
Next: Do not call slow downstream service inline.
Step 04 — Process idempotently
Why: Queue and worker can redeliver after crash.
Do: Claim event atomically, map to business key/version, apply transaction/outbox/provider idempotency, store terminal result and attempts.
Read the result: One logical effect despite concurrent duplicates.
Next: Treat terminal vs retryable errors distinctly.
Step 05 — Handle ordering and source of truth
Why: Delivery order may differ from creation order.
Do: Use event version/time cautiously; for mutable resources fetch current authoritative state when provider recommends; ignore stale transitions under state machine.
Read the result: Never let old event roll back confirmed newer state.
Next: Record decision.
Step 06 — Replay, reconcile, and observe
Why: Missed/failed events need safe recovery.
Do: Provide authenticated replay by event ID, dead-letter review, provider reconciliation sweep, alerts for signature/latency/failure/age/duplicates, and load tests.
Read the result: Replay cannot duplicate effect and every event reaches terminal disposition.
Next: Document retention.
Worked example
Evidence collected
- Signature is valid.
- Handler charges fulfillment inline and takes 12 seconds.
- Provider deadline is shorter, so it retries.
- No durable event-ID uniqueness exists.
Decision: Slow synchronous handling and missing idempotency cause duplicate fulfillment.
Actions taken
- Verified/stored/enqueued event then returned 2xx quickly.
- Added unique event ID and business payment ID with atomic fulfillment record.
- Replayed concurrent duplicates and simulated worker crash.
Why this example matters: Reliability comes from accepting duplicates safely, not assuming they will stop.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Raw-body signature, timestamp, environment, and rotation tests pass.
- Endpoint p95 acknowledgment fits provider deadline.
- Concurrent duplicate/replay creates one business effect.
- Out-of-order events cannot regress state.
- Retry/dead-letter/reconciliation reach terminal disposition.
- Secrets/payloads obey logging/retention policy.
Rollback or safe recovery
- Pause worker while still safely storing authenticated events.
- Return to prior handler/secret overlap configuration.
- Replay backlog only after idempotency/reconciliation is proven.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Valid events rejected after deploy | Body parser/order or secret environment changed. | Compare raw-byte middleware and active secret. |
| Queue grows | Worker/dependency slower than arrival. | Scale/bound, apply backpressure, inspect oldest age/failure class. |
| Same ID different body | Provider anomaly/attack/storage corruption. | Reject/alert; never overwrite original. |
| Replayed old event changes state | Ordering guard absent. | Use version/state transition or fetch current source. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Provider contract, endpoint/secret environment.
- Raw verification and replay policy.
- Durable event/idempotency state model.
- Worker retry/ordering/reconciliation behavior.
- Load/failure/replay metrics and runbook.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points