Backup and Restore Drill

Prove that critical files, databases, configurations, credentials, and service state can be restored within declared loss and downtime limits.

The result you're building

A recovery evidence package that maps business services to protected data, verifies backup integrity and isolation, restores into a clean environment, runs application checks, measures RPO/RTO, and records gaps and owners.

Use this guide when

• You depend on backups for business continuity, migration, ransomware, deletion, or client handoff.
• Backups exist but have not been restored recently.
• Configuration, identity, secrets, and dependencies matter as much as data files.

Do not use it as a substitute for

• Treating a green backup job as proof of recoverability.
• Testing by overwriting the only production copy or exposing production secrets in an insecure lab.

Before you change anything

• Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
• Service/data inventory, owners, dependencies, criticality, RPO, and RTO.
• Backup type, scope, schedule, retention, immutability, encryption, keys, and location.
• Last success, size, checksum, catalog, and restore instructions.
• Clean isolated recovery environment and replacement credentials.
• Restore timeline, integrity, application, reconciliation, failback, and deletion-replay evidence.

Understand the system before fixing it

Provenance is part of the record
A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.

Schema changes are product changes
Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.

Backup success and restore success are different
A job can omit files, corrupt catalogs, lose keys, capture inconsistent state, or depend on undocumented services while still reporting success.

Recovery includes identity and configuration
Data without DNS, certificates, secrets, schemas, versions, accounts, and runbooks may not produce a usable service.

Evidence-to-decision map

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

Step 01 — Map services to recovery objectives

Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: List critical outcomes, data, configs, identities, dependencies, owners, allowed loss, and downtime. Prioritize the minimum viable service.

Read the result: Every required component maps to RPO, RTO, and a backup source.

Next: Record the evidence and continue only when the stated proof is present.

Step 02 — Audit backup coverage and isolation

Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.

Do: Verify schedules, retention, immutability, encryption, key recovery, off-account/offline copies, monitoring, and production credential limits.

Read the result: A compromised production identity cannot destroy every recovery copy.

Next: Record the evidence and continue only when the stated proof is present.

Step 03 — Select a recovery point and verify bytes

Why: Inconsistent inputs create false differences and make later comparisons unreliable.

Do: Record backup IDs, time, scope, size, checksums, catalogs, logs, and known gaps. Preserve source backup unchanged.

Read the result: Chosen point is readable, complete by manifest, and inside RPO.

Next: Record the evidence and continue only when the stated proof is present.

Step 04 — Build a clean recovery environment

Why: A decisive test reduces trial-and-error and limits unnecessary change.

Do: Provision isolated network/accounts, pinned versions, capacity, secrets replacement, and safe DNS names. Keep production integrations disabled.

Read the result: Drill cannot send, charge, or overwrite production.

Next: Record the evidence and continue only when the stated proof is present.

Step 05 — Restore in dependency order

Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.

Do: Recover identity/config, database/storage, schemas, applications, queues, and indexes according to runbook while timing each stage.

Read the result: Each component passes integrity before the next dependency starts.

Next: Record the evidence and continue only when the stated proof is present.

Step 06 — Run business-level verification

Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.

Do: Test authentication, reads/writes, critical journeys, counts, constraints, file hashes, jobs, and permission boundaries; reconcile to the recovery point.

Read the result: Service is usable, not merely running.

Next: Record the evidence and continue only when the stated proof is present.

Step 07 — Record failover, failback, and gaps

Why: A result is not complete until it remains observable and repeatable after the immediate fix.

Do: Measure achieved RPO/RTO, replay deletions/events safely, document manual work, rotate exposed drill secrets, destroy lab state, and assign fixes.

Read the result: Evidence identifies pass/fail, remaining gaps, owner, deadline, and next drill.

Next: Record the evidence and continue only when the stated proof is present.

Operational worksheet

Evidence record

• Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
• Service/data inventory, owners, dependencies, criticality, RPO, and RTO.
• Backup type, scope, schedule, retention, immutability, encryption, keys, and location.
• Last success, size, checksum, catalog, and restore instructions.
• Clean isolated recovery environment and replacement credentials.
• Restore timeline, integrity, application, reconciliation, failback, and deletion-replay evidence.

Acceptance scoreboard

• Service inventory maps data, configuration, identity, dependencies, RPO, RTO, and owner.
• Backup copies are complete, encrypted, independently recoverable, and isolated from production compromise.
• Selected restore point and all required versions/checksums are recorded.
• Clean recovery environment cannot affect production.
• Integrity and business-journey tests pass after restore.
• Achieved RPO/RTO, deletion replay, failback, gaps, owners, and next drill are documented.

Minimum handoff record

• Versioned backup and restore drill scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Worked example

Evidence collected

• Dump files exist offsite.
• Checksums are valid.
• Key backup and recovery owner are undocumented.
• No restore drill has been performed.

Decision: The backup is operationally unusable. Establish independent key recovery and create a fresh protected backup after verifying the full restore.

Actions taken

• Recovered or replaced key management under documented owners.
• Restored into an isolated environment.
• Ran application and integrity tests.
• Added quarterly drill and key-loss scenario.

Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.

Verify, recover, and hand off

Completion tests

• A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
• Service inventory maps data, configuration, identity, dependencies, RPO, RTO, and owner.
• Backup copies are complete, encrypted, independently recoverable, and isolated from production compromise.
• Selected restore point and all required versions/checksums are recorded.
• Clean recovery environment cannot affect production.
• Integrity and business-journey tests pass after restore.
• Achieved RPO/RTO, deletion replay, failback, gaps, owners, and next drill are documented.

Rollback or safe recovery

• Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
• Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
• Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
• Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.

If the expected result does not appear

Reusable handoff record

• Versioned backup and restore drill scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Agent delivery contract

Required inputs

Agent refusal and escalation rules

• Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
• Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
• Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
• Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.

Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.

Official reference starting points

• https://www.cisa.gov/news-events/news/data-backup-options
• https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final
• https://www.postgresql.org/docs/current/backup.html