The result you are building

Finished Result:

An event contract and consumer design that records immutable identity, validates schema, processes idempotently, handles ordering and retries, quarantines poison events, and reconciles producer-to-consumer outcomes.

Use this guide when

Webhooks, queues, streams, or event buses drive agent and application work.
Events may be duplicated, delayed, reordered, or redelivered.
Consumers create messages, payments, database writes, or other side effects.

Do not use it as a substitute for

Assuming 'exactly once' marketing removes application-level idempotency.
Deleting poison messages or dead letters without preserving evidence and owner.

Before you change anything

Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.

Event type/version, producer, stable ID, subject, occurrence/publish time, and schema.
Broker retention, ordering key, partition, delivery, ack, visibility, and redrive semantics.
Consumer idempotency store, side-effect transaction boundary, and replay policy.
Retry classification, backoff, dead-letter owner, and privacy/retention.
Duplicate, reorder, delay, poison, crash, replay, and reconciliation test evidence.

Stop Before Proceeding:

Stop a consumer that cannot distinguish duplicate from new work or whose dead-letter backlog contains unreviewed high-impact events. Preserve the event and processing record before redrive.

Understand the system before fixing it

Provenance is part of the record A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.

Schema changes are product changes Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.

Delivery and processing are separate A broker can deliver an event while the consumer fails before or after committing. Record receipt, processing, side effect, and acknowledgment separately.

Ordering is scoped, not global When order matters, choose an entity/aggregate key and handle version gaps. Global serialization often destroys capacity without solving cross-source causality.

Evidence-to-decision map

Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.

   Evidence               Likely layer     First decisive check           What the result means

   Same webhook           Idempotency      Replay event ID and business   Consumer commits side effect without atomic deduplication.
   sends two payments                      key

   Newer update           Ordering/versi   Compare entity version and     Consumer applies arrival order without stale-event rejection.
   overwritten by older   on               event time

   Message retries        Poison           Validate schema and inspect    Permanent errors are treated as transient.
   forever                classification   deterministic failure

   Acked event has no     Atomicity        Trace ack relative to state    Acknowledgment occurred before durable processing outcome.
   result                                  commit

   Backlog grows with     Capacity         Compare arrival/service rate   Consumer throughput or dependency is below required rate.
   normal traffic                          and oldest age

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

01 Define the event contract Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: Use stable event ID, type, version, source, subject, occurred/published time, correlation, data schema, classification, and compatibility policy.

Read the result: Valid and invalid examples pass deterministic schema tests.