SAYLOR Process events under at-least-once delivery without silent loss, duplicate side effects, poison-message loops, ordering errors, or unbounded backlog.
The result you're building
An event contract and consumer design that records immutable identity, validates schema, processes idempotently, handles ordering and retries, quarantines poison events, and reconciles producer-to-consumer outcomes.
Use this guide when
- Webhooks, queues, streams, or event buses drive agent and application work.
- Events may be duplicated, delayed, reordered, or redelivered.
- Consumers create messages, payments, database writes, or other side effects.
Do not use it as a substitute for
- Assuming 'exactly once' marketing removes application-level idempotency.
- Deleting poison messages or dead letters without preserving evidence and owner.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Event type/version, producer, stable ID, subject, occurrence/publish time, and schema.
- Broker retention, ordering key, partition, delivery, ack, visibility, and redrive semantics.
- Consumer idempotency store, side-effect transaction boundary, and replay policy.
- Retry classification, backoff, dead-letter owner, and privacy/retention.
- Duplicate, reorder, delay, poison, crash, replay, and reconciliation test evidence.
Understand the system before fixing it
Provenance is part of the record
A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.
Schema changes are product changes
Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.
Delivery and processing are separate
A broker can deliver an event while the consumer fails before or after committing. Record receipt, processing, side effect, and acknowledgment separately.
Ordering is scoped, not global
When order matters, choose an entity/aggregate key and handle version gaps. Global serialization often destroys capacity without solving cross-source causality.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Same webhook sends two payments | Idempotency | Replay event ID and business key | Consumer commits side effect without atomic deduplication. |
| Newer update overwritten by older | Ordering/version | Compare entity version and event time | Consumer applies arrival order without stale-event rejection. |
| Message retries forever | Poison classification | Validate schema and inspect deterministic failure | Permanent errors are treated as transient. |
| Acked event has no result | Atomicity | Trace ack relative to state commit | Acknowledgment occurred before durable processing outcome. |
| Backlog grows with normal traffic | Capacity | Compare arrival/service rate and oldest age | Consumer throughput or dependency is below required rate. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Define the event contract
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Use stable event ID, type, version, source, subject, occurred/published time, correlation, data schema, classification, and compatibility policy.
Read the result: Valid and invalid examples pass deterministic schema tests.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Choose delivery and ordering semantics
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Document broker ack, redelivery, visibility, retention, partition/order key, batch, and maximum delay. Match ordering to the real aggregate.
Read the result: Failure and rebalance behavior is understood before consumer code.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Build idempotent processing
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Reserve event/business key atomically, compare payload hash, store terminal result, and ensure retries return the same outcome. Coordinate database and external side effects.
Read the result: Duplicate event cannot duplicate effect; conflicting reuse is rejected.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Reject stale and invalid state transitions
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Track entity version or monotonic transition, handle gaps, and quarantine impossible or out-of-order updates rather than overwriting current state.
Read the result: Forced reorder preserves the correct final entity state.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Classify retry and dead letters
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Retry transient dependencies with capped jittered backoff; send invalid, unauthorized, expired, or repeatedly failing events to a visible quarantine with owner.
Read the result: Poison events stop consuming the hot path and retain complete evidence.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Make replay safe
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Version consumer logic, snapshot dependencies where needed, define historical side-effect policy, and require approval for redrive of high-impact events.
Read the result: Replaying a known range produces expected state without external duplication.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Reconcile end to end
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Compare producer counts/IDs, broker state, consumer outcomes, side effects, dead letters, and backlog age. Test crash before/after commit and ack.
Read the result: Every accepted event is terminal, pending within SLA, or visibly quarantined.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Event type/version, producer, stable ID, subject, occurrence/publish time, and schema.
- Broker retention, ordering key, partition, delivery, ack, visibility, and redrive semantics.
- Consumer idempotency store, side-effect transaction boundary, and replay policy.
- Retry classification, backoff, dead-letter owner, and privacy/retention.
- Duplicate, reorder, delay, poison, crash, replay, and reconciliation test evidence.
Acceptance scoreboard
- Event schema, identity, source, time, version, and compatibility are explicit.
- Broker delivery, ack, retention, ordering, and redrive behavior are documented.
- Duplicate and conflicting events are handled atomically at the side-effect boundary.
- Stale or out-of-order updates cannot regress entity state.
- Permanent failures enter owned quarantine; transient retries are bounded.
- Producer, broker, consumer, side effect, dead letter, and backlog reconcile.
Minimum handoff record
- Versioned queue and event delivery reliability scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- Provider event ID is stable.
- Consumer stores deduplication after the charge.
- Crash occurs before dedup write and ack.
- Second delivery calls the charge API again with a new key.
Decision: The idempotency boundary is too late. Reserve the event and use the same business idempotency key for the external charge.
Actions taken
- Atomically reserved provider event and payload hash.
- Derived charge key from order and event.
- Stored charge result before acknowledgment.
- Added crash-window replay tests.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Event schema, identity, source, time, version, and compatibility are explicit.
- Broker delivery, ack, retention, ordering, and redrive behavior are documented.
- Duplicate and conflicting events are handled atomically at the side-effect boundary.
- Stale or out-of-order updates cannot regress entity state.
- Permanent failures enter owned quarantine; transient retries are bounded.
- Producer, broker, consumer, side effect, dead letter, and backlog reconcile.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Same webhook sends two payments | Consumer commits side effect without atomic deduplication. | Replay event ID and business key |
| Newer update overwritten by older | Consumer applies arrival order without stale-event rejection. | Compare entity version and event time |
| Message retries forever | Permanent errors are treated as transient. | Validate schema and inspect deterministic failure |
| Acked event has no result | Acknowledgment occurred before durable processing outcome. | Trace ack relative to state commit |
Reusable handoff record
- Versioned queue and event delivery reliability scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points