SAYLOR The result you are building
Finished Result:
A recovery evidence package that maps business services to protected data, verifies backup integrity and isolation, restores into a clean environment, runs application checks, measures RPO/RTO, and records gaps and owners.
Use this guide when
- You depend on backups for business continuity, migration, ransomware, deletion, or client handoff.
- Backups exist but have not been restored recently.
- Configuration, identity, secrets, and dependencies matter as much as data files.
Do not use it as a substitute for
- Treating a green backup job as proof of recoverability.
- Testing by overwriting the only production copy or exposing production secrets in an insecure lab.
Before you change anything
Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Service/data inventory, owners, dependencies, criticality, RPO, and RTO.
- Backup type, scope, schedule, retention, immutability, encryption, keys, and location.
- Last success, size, checksum, catalog, and restore instructions.
- Clean isolated recovery environment and replacement credentials.
- Restore timeline, integrity, application, reconciliation, failback, and deletion-replay evidence.
Stop Before Proceeding:
Do not modify production during a drill without an approved failover plan. Stop if the only backup, decryption key, or authoritative recovery record would be placed at risk.
Understand the system before fixing it
Provenance is part of the record A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.
Schema changes are product changes Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.
Backup success and restore success are different A job can omit files, corrupt catalogs, lose keys, capture inconsistent state, or depend on undocumented services while still reporting success.
Recovery includes identity and configuration Data without DNS, certificates, secrets, schemas, versions, accounts, and runbooks may not produce a usable service.
Evidence-to-decision map
Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.
Evidence Likely layer First decisive check What the result means
Backup verifies but Dependency/c Compare recovery manifest to Configuration, secrets, binary version, schema, or external
app will not start onfig service inventory dependency is missing.
Newest restore has Consistency Check snapshot/transaction Files or databases were captured without application-consistent
inconsistent rows boundary and logs coordination.
Restore exceeds RTO Procedure/cap Break timeline into transfer, Throughput, manual steps, or index rebuild controls downtime.
acity decrypt, restore, rebuild, verify
Ransomware can Isolation Test production credential Backups are not sufficiently isolated or immutable.
delete backups permissions against backup
vault
Deleted data Lifecycle Replay tombstones/retention Recovery procedure ignores post-backup deletions and
reappears after restore corrections.Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
01 Map services to recovery objectives Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: List critical outcomes, data, configs, identities, dependencies, owners, allowed loss, and downtime. Prioritize the minimum viable service.
Read the result: Every required component maps to RPO, RTO, and a backup source.
Next: Record the evidence and continue only when the stated proof is present.
02 Audit backup coverage and isolation Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Verify schedules, retention, immutability, encryption, key recovery, off-account/offline copies, monitoring, and production credential limits.
Read the result: A compromised production identity cannot destroy every recovery copy.
Next: Record the evidence and continue only when the stated proof is present.
03 Select a recovery point and verify bytes Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Record backup IDs, time, scope, size, checksums, catalogs, logs, and known gaps. Preserve source backup unchanged.
Read the result: Chosen point is readable, complete by manifest, and inside RPO.
Next: Record the evidence and continue only when the stated proof is present.
04 Build a clean recovery environment Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Provision isolated network/accounts, pinned versions, capacity, secrets replacement, and safe DNS names. Keep production integrations disabled.
Read the result: Drill cannot send, charge, or overwrite production.
Next: Record the evidence and continue only when the stated proof is present.
Procedure continued 05 Restore in dependency order Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Recover identity/config, database/storage, schemas, applications, queues, and indexes according to runbook while timing each stage.
Read the result: Each component passes integrity before the next dependency starts.
Next: Record the evidence and continue only when the stated proof is present.
06 Run business-level verification Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Test authentication, reads/writes, critical journeys, counts, constraints, file hashes, jobs, and permission boundaries; reconcile to the recovery point.
Read the result: Service is usable, not merely running.
Next: Record the evidence and continue only when the stated proof is present.
07 Record failover, failback, and gaps Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Measure achieved RPO/RTO, replay deletions/events safely, document manual work, rotate exposed drill secrets, destroy lab state, and assign fixes.
Read the result: Evidence identifies pass/fail, remaining gaps, owner, deadline, and next drill.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet Evidence record Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Service/data inventory, owners, dependencies, criticality, RPO, and RTO.
- Backup type, scope, schedule, retention, immutability, encryption, keys, and location.
- Last success, size, checksum, catalog, and restore instructions.
- Clean isolated recovery environment and replacement credentials.
- Restore timeline, integrity, application, reconciliation, failback, and deletion-replay evidence.
Acceptance scoreboard
- Service inventory maps data, configuration, identity, dependencies, RPO, RTO, and owner.
- Backup copies are complete, encrypted, independently recoverable, and isolated from production compromise.
- Selected restore point and all required versions/checksums are recorded.
- Clean recovery environment cannot affect production.
- Integrity and business-journey tests pass after restore.
- Achieved RPO/RTO, deletion replay, failback, gaps, owners, and next drill are documented.
Decision rule SHIP / AUTOMATE GATE Proceed only when every required acceptance check is supported by direct evidence, rollback is available, and the remaining risk is explicitly owned. Unknown is not a pass.
Minimum handoff record
- Versioned backup and restore drill scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Starting Problem:
A client has nightly database dumps, but the restore fails because the encryption key lived only on the lost server.
Evidence collected
- Dump files exist offsite.
- Checksums are valid.
- Key backup and recovery owner are undocumented.
- No restore drill has been performed.
Decision The backup is operationally unusable. Establish independent key recovery and create a fresh protected backup after verifying the full restore.
Actions taken
- Recovered or replaced key management under documented owners.
- Restored into an isolated environment.
- Ran application and integrity tests.
- Added quarterly drill and key-loss scenario.
Proof Of Completion:
A clean operator with authorized recovery access restores within RPO/RTO without the original server and proves critical business journeys.
Why this example matters The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Service inventory maps data, configuration, identity, dependencies, RPO, RTO, and owner.
- Backup copies are complete, encrypted, independently recoverable, and isolated from production compromise.
- Selected restore point and all required versions/checksums are recorded.
- Clean recovery environment cannot affect production.
- Integrity and business-journey tests pass after restore.
- Achieved RPO/RTO, deletion replay, failback, gaps, owners, and next drill are documented.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear What happened What it usually means Next safe move
Backup verifies but app will Configuration, secrets, binary version, Compare recovery manifest to service inventory not start schema, or external dependency is missing.
Newest restore has Files or databases were captured Check snapshot/transaction boundary and logs inconsistent rows without application-consistent coordination.
Restore exceeds RTO Throughput, manual steps, or index Break timeline into transfer, decrypt, restore, rebuild, verify rebuild controls downtime.
Ransomware can delete Backups are not sufficiently isolated or Test production credential permissions against backup vault backups immutable.
Reusable handoff record
- Versioned backup and restore drill scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
Field Type Requirement
target object Versioned environment, resource, identity, or workflow being evaluated.
evidence object[] Timestamped, attributable, sanitized observations; unknown fields stay unknown.
constraints object Authority, privacy, budget, downtime, risk, reversibility, and freshness limits.
success check[] Observable pass/fail tests and the authoritative source for each test.
Returned output
Field Type Requirement
diagnosis object Likely layer, supporting and conflicting evidence, alternatives, and confidence.
plan step[] Ordered bounded actions with owner, risk, expected proof, and stop condition.
verification check[] Observed pass/fail/unknown results, not inferred success from command exit alone.
handoff object Sanitized evidence record, recovery state, remaining risk, and next review trigger.
Agent refusal and escalation rules
•
Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
•
Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
•
Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
•
Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule
Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low
confidence; a controlled test that isolates the layer and passes verification can support high confidence.Official reference starting points
- https://www.cisa.gov/news-events/news/data-backup-options
- https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final
- https://www.postgresql.org/docs/current/backup.html