CI/CD Failed Deployment Recovery

Recover a failed release by identifying build, artifact, configuration, migration, rollout, health, routing, or dependency failure while preserving a known-good version.

The result you're building

A release pipeline and recovery runbook with immutable artifacts, environment promotion, preflight and health gates, staged rollout, migration compatibility, one-click stop/rollback, and end-to-end verification.

Use this guide when

• A deploy fails, hangs, rolls back, or appears successful while the service is broken.
• Different environments build different artifacts.
• Database, config, feature flags, or routing changes accompany code.

Do not use it as a substitute for

• Redeploying random commits or rebuilding the same version with changed dependencies.
• Rolling back application code blindly after an irreversible migration or accepted new-format writes.

Before you change anything

• Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
• Pipeline/run/release IDs, commit/tag, artifact/image digest, builder, and dependency lock.
• Environment config/secret/flag versions and change approvals.
• Migration state, compatibility window, backup/restore proof.
• Build/test/security/deploy logs, rollout events, health, traces, metrics, and user journey.
• Last-known-good release, traffic/routing state, rollback/roll-forward, and incident record.

Understand the system before fixing it

Observe before mutating
Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.

Recovery must be exercised
A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.

Build once, promote the same bytes
Rebuilding for production can change dependencies, timestamps, base images, or generated code. Promote an immutable verified artifact with environment configuration outside it.

Platform success is not product success
A deployment can reach 'ready' while auth, database, payments, DNS, or critical user journeys fail. Use layered and end-to-end checks.

Evidence-to-decision map

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

Step 01 — Freeze and identify the release

Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: Record pipeline, commit, artifact digest, environment, config/secret/flag versions, migration, rollout, routing, and last-known-good. Pause automatic promotion.

Read the result: There is one exact failed release and one verified recovery target.

Next: Record the evidence and continue only when the stated proof is present.

Step 02 — Classify the failing stage

Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.

Do: Separate source, dependency, build, test, scan, artifact, deploy control plane, startup, health, routing, database, dependency, and user-journey evidence.

Read the result: The first failing layer is isolated before another deploy.

Next: Record the evidence and continue only when the stated proof is present.

Step 03 — Protect data and compatibility

Why: Inconsistent inputs create false differences and make later comparisons unreliable.

Do: Inspect migrations, backups, new writes, queues, and old/new version compatibility. Decide rollback versus roll forward based on representational safety.

Read the result: Recovery will not discard accepted data or create incompatible writers.

Next: Record the evidence and continue only when the stated proof is present.

Step 04 — Recover with the narrowest action

Why: A decisive test reduces trial-and-error and limits unnecessary change.

Do: Stop rollout, shift traffic to known-good compatible release, correct config/secret/route, or deploy a minimal forward fix. Retain failed artifacts and logs.

Read the result: User impact falls and authoritative state remains consistent.

Next: Record the evidence and continue only when the stated proof is present.

Step 05 — Verify layered health

Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.

Do: Check platform status, process, readiness, logs, dependency calls, database, auth, critical API, browser journey, observability, and external routing.

Read the result: The requested product outcome works, not just the deployment command.

Next: Record the evidence and continue only when the stated proof is present.

Step 06 — Repair pipeline controls

Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.

Do: Pin dependencies/base images, build immutable artifacts once, sign/attest where appropriate, validate config schema, enforce migration and canary gates, and retain provenance.

Read the result: The failure class is blocked or detected before broad traffic.

Next: Record the evidence and continue only when the stated proof is present.

Step 07 — Rehearse stop and rollback

Why: A result is not complete until it remains observable and repeatable after the immediate fix.

Do: Inject failed startup, bad health, config error, dependency outage, migration incompatibility, and partial rollout. Measure detection and recovery.

Read the result: Pipeline stops automatically and operator recovery meets the target.

Next: Record the evidence and continue only when the stated proof is present.

Operational worksheet

Evidence record

• Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
• Pipeline/run/release IDs, commit/tag, artifact/image digest, builder, and dependency lock.
• Environment config/secret/flag versions and change approvals.
• Migration state, compatibility window, backup/restore proof.
• Build/test/security/deploy logs, rollout events, health, traces, metrics, and user journey.
• Last-known-good release, traffic/routing state, rollback/roll-forward, and incident record.

Acceptance scoreboard

• Release identity includes immutable artifact digest and exact environment configuration versions.
• First failing layer is isolated from build through user journey.
• Migration/data compatibility determines rollback versus roll forward.
• Recovery preserves failed evidence and last-known-good service.
• Layered and end-to-end checks prove actual product behavior.
• Pipeline provenance, config validation, canary, stop, and rollback failure tests pass.

Minimum handoff record

• Versioned ci/cd failed deployment recovery scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Worked example

Evidence collected

• Container readiness checks only /health without dependencies.
• Artifact digest matches staging.
• Production secret uses old key name.
• No configuration schema check exists.

Decision: The artifact is valid; the production environment contract and health gate are incomplete. Restore compatible secret mapping or roll forward safely.

Actions taken

• Paused rollout and routed to last-known-good.
• Validated production config against a schema without exposing values.
• Added authenticated dependency-aware canary.
• Promoted the same artifact through staging and production.

Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.

Verify, recover, and hand off

Completion tests

• A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
• Release identity includes immutable artifact digest and exact environment configuration versions.
• First failing layer is isolated from build through user journey.
• Migration/data compatibility determines rollback versus roll forward.
• Recovery preserves failed evidence and last-known-good service.
• Layered and end-to-end checks prove actual product behavior.
• Pipeline provenance, config validation, canary, stop, and rollback failure tests pass.

Rollback or safe recovery

• Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
• Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
• Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
• Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.

If the expected result does not appear

Reusable handoff record

• Versioned ci/cd failed deployment recovery scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Agent delivery contract

Required inputs

Agent refusal and escalation rules

• Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
• Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
• Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
• Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.

Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.

Official reference starting points

• https://slsa.dev/
• https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions
• https://kubernetes.io/docs/concepts/workloads/controllers/deployment/