SAYLOR Build a defensible incident chronology from logs, traces, alerts, deployments, user reports, and authoritative system state without confusing correlation with cause.
The result you're building
A normalized timeline and causal analysis that preserves original evidence, states clock uncertainty, separates trigger/root cause/contributing factors/impact, tests competing explanations, and produces owned corrective actions.
Use this guide when
- A service, security, data, trading, or automation incident has unclear sequence or cause.
- Different systems use different clocks and identifiers.
- You need a post-incident record that improves controls instead of blaming a person.
Do not use it as a substitute for
- Starting with a favored root cause and selecting matching events.
- Editing or deleting original logs while building the narrative.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Incident scope, detection time, impact, owners, and evidence-preservation authority.
- Raw logs/traces/audit/deploy/config/alerts/tickets with source, time zone, clock offset, and retention.
- Correlation IDs, identities, hosts, versions, request/event/business keys.
- Known-good baseline, changes, dependency state, and recovery actions.
- Competing hypotheses, decisive tests, gaps, and confidence.
Understand the system before fixing it
Provenance is part of the record
A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.
Schema changes are product changes
Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.
Timeline order requires clock context
Wall clocks can drift and time zones differ. Use monotonic durations, trace relationships, provider times, and measured offsets where possible.
Trigger and root cause are not the same
A deployment may trigger an outage while unsafe coupling, absent limits, or bad recovery design is the deeper cause.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Logs disagree on event order | Clock/correlation | Compare offsets, trace parents, and provider timestamps | Wall-clock ordering alone is unreliable. |
| Error begins after deployment | Change correlation | Reproduce or roll back under controlled observation | Deployment is a candidate trigger, not yet proven root cause. |
| Only one tenant affected | Scope/data | Compare routing, feature, identity, and data slice | Failure likely depends on tenant-specific state or policy. |
| Restart fixes issue but cause unknown | Volatile state | Inspect resource, queue, connection, and version evidence | Recovery erased symptoms without isolating cause. |
| No evidence for a claimed action | Observability | Search authoritative audit/side-effect record | The action is unknown, not safely assumed to have happened or not happened. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Declare incident scope and preserve evidence
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Record start estimate, affected outcomes, systems, data sensitivity, owners, and authority. Snapshot logs, traces, configs, deployments, queues, balances, and volatile state with hashes.
Read the result: Evidence copies are attributable and originals remain protected.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Normalize time and identity
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Convert to a declared time base while retaining original timestamps, time zones, offsets, and uncertainty. Map hosts, services, users, requests, events, and releases to stable IDs.
Read the result: Cross-system events can be compared without hiding clock uncertainty.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Build a fact-only chronology
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: List observations with source and confidence before interpretation. Separate alerts, symptoms, actions, state changes, and verified effects.
Read the result: Every timeline line points to evidence or is labeled reported/estimated.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Bound impact and exposure
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Measure who/what was affected, duration, failed/succeeded operations, money/data exposure, and unknowns. Reconcile side effects independently.
Read the result: Impact statement is quantitative and does not extrapolate beyond evidence.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Test competing hypotheses
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: For each cause candidate, list predicted evidence, supporting/conflicting observations, and decisive safe tests. Avoid single-factor stories.
Read the result: Chosen explanation outperforms alternatives and states residual uncertainty.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Separate cause categories
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Document trigger, direct technical cause, contributing conditions, detection/recovery gaps, and organizational/systemic factors without blame language.
Read the result: Corrective actions address more than the triggering event.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Verify fixes and institutionalize learning
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Add reproduction/regression tests, observability, limits, runbook, owner, due date, and success metric. Re-run failure scenario and recovery.
Read the result: Actions reduce recurrence or impact and are tracked to evidence-based closure.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Incident scope, detection time, impact, owners, and evidence-preservation authority.
- Raw logs/traces/audit/deploy/config/alerts/tickets with source, time zone, clock offset, and retention.
- Correlation IDs, identities, hosts, versions, request/event/business keys.
- Known-good baseline, changes, dependency state, and recovery actions.
- Competing hypotheses, decisive tests, gaps, and confidence.
Acceptance scoreboard
- Original evidence is preserved with source, hash, time, offset, and owner.
- Timeline retains original timestamps and states clock uncertainty.
- Impact reconciles authoritative side effects, affected scope, duration, and unknowns.
- Competing hypotheses include supporting, conflicting, and decisive evidence.
- Trigger, root cause, contributing factors, and recovery gaps are distinct.
- Corrective actions have owners, tests, metrics, dates, and verified closure.
Minimum handoff record
- Versioned incident timeline and root-cause evidence scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- Bot host clock is 47 seconds fast.
- Exchange order IDs show two accepted requests.
- Retries use new client IDs.
- RPC slowdown began before duplicates, but duplicate logic is local.
Decision: The slowdown triggered the path; missing stable idempotency and clock discipline caused duplicate orders. Both belong in the analysis at different layers.
Actions taken
- Normalized using exchange/provider time and measured host offset.
- Reconciled every order and balance.
- Bound retries to stable order intent IDs.
- Added clock, timeout, and duplicate regression tests.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Original evidence is preserved with source, hash, time, offset, and owner.
- Timeline retains original timestamps and states clock uncertainty.
- Impact reconciles authoritative side effects, affected scope, duration, and unknowns.
- Competing hypotheses include supporting, conflicting, and decisive evidence.
- Trigger, root cause, contributing factors, and recovery gaps are distinct.
- Corrective actions have owners, tests, metrics, dates, and verified closure.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Logs disagree on event order | Wall-clock ordering alone is unreliable. | Compare offsets, trace parents, and provider timestamps |
| Error begins after deployment | Deployment is a candidate trigger, not yet proven root cause. | Reproduce or roll back under controlled observation |
| Only one tenant affected | Failure likely depends on tenant-specific state or policy. | Compare routing, feature, identity, and data slice |
| Restart fixes issue but cause unknown | Recovery erased symptoms without isolating cause. | Inspect resource, queue, connection, and version evidence |
Reusable handoff record
- Versioned incident timeline and root-cause evidence scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points