SAYLOR The result you are building
Finished Result:
A normalized timeline and causal analysis that preserves original evidence, states clock uncertainty, separates trigger/root cause/contributing factors/impact, tests competing explanations, and produces owned corrective actions.
Use this guide when
- A service, security, data, trading, or automation incident has unclear sequence or cause.
- Different systems use different clocks and identifiers.
- You need a post-incident record that improves controls instead of blaming a person.
Do not use it as a substitute for
- Starting with a favored root cause and selecting matching events.
- Editing or deleting original logs while building the narrative.
Before you change anything
Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Incident scope, detection time, impact, owners, and evidence-preservation authority.
- Raw logs/traces/audit/deploy/config/alerts/tickets with source, time zone, clock offset, and retention.
- Correlation IDs, identities, hosts, versions, request/event/business keys.
- Known-good baseline, changes, dependency state, and recovery actions.
- Competing hypotheses, decisive tests, gaps, and confidence.
Stop Before Proceeding:
Stop destructive cleanup, log rotation, redeployment, or clock changes until volatile evidence and current state are preserved, unless immediate safety requires isolation first.
Understand the system before fixing it
Provenance is part of the record A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.
Schema changes are product changes Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.
Timeline order requires clock context Wall clocks can drift and time zones differ. Use monotonic durations, trace relationships, provider times, and measured offsets where possible.
Trigger and root cause are not the same A deployment may trigger an outage while unsafe coupling, absent limits, or bad recovery design is the deeper cause.
Evidence-to-decision map
Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.
Evidence Likely layer First decisive check What the result means
Logs disagree on Clock/correlati Compare offsets, trace parents, Wall-clock ordering alone is unreliable.
event order on and provider timestamps
Error begins after Change Reproduce or roll back under Deployment is a candidate trigger, not yet proven root cause.
deployment correlation controlled observation
Only one tenant Scope/data Compare routing, feature, Failure likely depends on tenant-specific state or policy.
affected identity, and data slice
Restart fixes issue Volatile state Inspect resource, queue, Recovery erased symptoms without isolating cause.
but cause unknown connection, and version
evidence
No evidence for a Observability Search authoritative The action is unknown, not safely assumed to have happened or
claimed action audit/side-effect record not happened.Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
01 Declare incident scope and preserve evidence Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Record start estimate, affected outcomes, systems, data sensitivity, owners, and authority. Snapshot logs, traces, configs, deployments, queues, balances, and volatile state with hashes.
Read the result: Evidence copies are attributable and originals remain protected.
Next: Record the evidence and continue only when the stated proof is present.
02 Normalize time and identity Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Convert to a declared time base while retaining original timestamps, time zones, offsets, and uncertainty. Map hosts, services, users, requests, events, and releases to stable IDs.
Read the result: Cross-system events can be compared without hiding clock uncertainty.
Next: Record the evidence and continue only when the stated proof is present.
03 Build a fact-only chronology Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: List observations with source and confidence before interpretation. Separate alerts, symptoms, actions, state changes, and verified effects.
Read the result: Every timeline line points to evidence or is labeled reported/estimated.
Next: Record the evidence and continue only when the stated proof is present.
04 Bound impact and exposure Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Measure who/what was affected, duration, failed/succeeded operations, money/data exposure, and unknowns. Reconcile side effects independently.
Read the result: Impact statement is quantitative and does not extrapolate beyond evidence.
Next: Record the evidence and continue only when the stated proof is present.
Procedure continued 05 Test competing hypotheses Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: For each cause candidate, list predicted evidence, supporting/conflicting observations, and decisive safe tests. Avoid single-factor stories.
Read the result: Chosen explanation outperforms alternatives and states residual uncertainty.
Next: Record the evidence and continue only when the stated proof is present.
06 Separate cause categories Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Document trigger, direct technical cause, contributing conditions, detection/recovery gaps, and organizational/systemic factors without blame language.
Read the result: Corrective actions address more than the triggering event.
Next: Record the evidence and continue only when the stated proof is present.
07 Verify fixes and institutionalize learning Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Add reproduction/regression tests, observability, limits, runbook, owner, due date, and success metric. Re-run failure scenario and recovery.
Read the result: Actions reduce recurrence or impact and are tracked to evidence-based closure.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet Evidence record Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Incident scope, detection time, impact, owners, and evidence-preservation authority.
- Raw logs/traces/audit/deploy/config/alerts/tickets with source, time zone, clock offset, and retention.
- Correlation IDs, identities, hosts, versions, request/event/business keys.
- Known-good baseline, changes, dependency state, and recovery actions.
- Competing hypotheses, decisive tests, gaps, and confidence.
Acceptance scoreboard
- Original evidence is preserved with source, hash, time, offset, and owner.
- Timeline retains original timestamps and states clock uncertainty.
- Impact reconciles authoritative side effects, affected scope, duration, and unknowns.
- Competing hypotheses include supporting, conflicting, and decisive evidence.
- Trigger, root cause, contributing factors, and recovery gaps are distinct.
- Corrective actions have owners, tests, metrics, dates, and verified closure.
Decision rule SHIP / AUTOMATE GATE Proceed only when every required acceptance check is supported by direct evidence, rollback is available, and the remaining risk is explicitly owned. Unknown is not a pass.
Minimum handoff record
- Versioned incident timeline and root-cause evidence scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Starting Problem:
A trading bot sends duplicate orders during an RPC slowdown, and logs from the bot and exchange show conflicting times.
Evidence collected
- Bot host clock is 47 seconds fast.
- Exchange order IDs show two accepted requests.
- Retries use new client IDs.
- RPC slowdown began before duplicates, but duplicate logic is local.
Decision The slowdown triggered the path; missing stable idempotency and clock discipline caused duplicate orders. Both belong in the analysis at different layers.
Actions taken
- Normalized using exchange/provider time and measured host offset.
- Reconciled every order and balance.
- Bound retries to stable order intent IDs.
- Added clock, timeout, and duplicate regression tests.
Proof Of Completion:
Replayed slow responses create one logical order, logs correlate across systems, and reconciliation proves final positions and costs.
Why this example matters The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Original evidence is preserved with source, hash, time, offset, and owner.
- Timeline retains original timestamps and states clock uncertainty.
- Impact reconciles authoritative side effects, affected scope, duration, and unknowns.
- Competing hypotheses include supporting, conflicting, and decisive evidence.
- Trigger, root cause, contributing factors, and recovery gaps are distinct.
- Corrective actions have owners, tests, metrics, dates, and verified closure.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear What happened What it usually means Next safe move
Logs disagree on event order Wall-clock ordering alone is unreliable. Compare offsets, trace parents, and provider timestamps
Error begins after deployment Deployment is a candidate trigger, not Reproduce or roll back under controlled observation yet proven root cause.
Only one tenant affected Failure likely depends on Compare routing, feature, identity, and data slice tenant-specific state or policy.
Restart fixes issue but cause Recovery erased symptoms without Inspect resource, queue, connection, and version evidence unknown isolating cause.
Reusable handoff record
- Versioned incident timeline and root-cause evidence scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
Field Type Requirement
target object Versioned environment, resource, identity, or workflow being evaluated.
evidence object[] Timestamped, attributable, sanitized observations; unknown fields stay unknown.
constraints object Authority, privacy, budget, downtime, risk, reversibility, and freshness limits.
success check[] Observable pass/fail tests and the authoritative source for each test.
Returned output
Field Type Requirement
diagnosis object Likely layer, supporting and conflicting evidence, alternatives, and confidence.
plan step[] Ordered bounded actions with owner, risk, expected proof, and stop condition.
verification check[] Observed pass/fail/unknown results, not inferred success from command exit alone.
handoff object Sanitized evidence record, recovery state, remaining risk, and next review trigger.
Agent refusal and escalation rules
•
Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
•
Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
•
Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
•
Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule
Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low
confidence; a controlled test that isolates the layer and passes verification can support high confidence.Official reference starting points
- https://csrc.nist.gov/pubs/sp/800/61/r3/final
- https://www.cisa.gov/news-events/news/federal-government-cybersecurity-incident-and-vulnerability-response-playbooks
- https://opentelemetry.io/docs/concepts/signals/traces/