CI/CD Failed Deployment Recovery

The result you are building

Finished Result:

A release pipeline and recovery runbook with immutable artifacts, environment promotion, preflight and health gates, staged rollout, migration compatibility, one-click stop/rollback, and end-to-end verification.

Use this guide when

• A deploy fails, hangs, rolls back, or appears successful while the service is broken.
• Different environments build different artifacts.
• Database, config, feature flags, or routing changes accompany code.

Do not use it as a substitute for

• Redeploying random commits or rebuilding the same version with changed dependencies.
• Rolling back application code blindly after an irreversible migration or accepted new-format writes.

Before you change anything

Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.

• Pipeline/run/release IDs, commit/tag, artifact/image digest, builder, and dependency lock.
• Environment config/secret/flag versions and change approvals.
• Migration state, compatibility window, backup/restore proof.
• Build/test/security/deploy logs, rollout events, health, traces, metrics, and user journey.
• Last-known-good release, traffic/routing state, rollback/roll-forward, and incident record.

Stop Before Proceeding:

Stop new rollout and preserve release evidence when health gates fail, artifact identity is uncertain, or data compatibility prevents a safe rollback. Do not delete failed resources before the cause and recovery state are captured.

Understand the system before fixing it

Observe before mutating Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.

Recovery must be exercised A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.

Build once, promote the same bytes Rebuilding for production can change dependencies, timestamps, base images, or generated code. Promote an immutable verified artifact with environment configuration outside it.

Platform success is not product success A deployment can reach 'ready' while auth, database, payments, DNS, or critical user journeys fail. Use layered and end-to-end checks.

Evidence-to-decision map

Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.

   Evidence                Likely layer     First decisive check             What the result means

   Build passes,           Runtime/confi    Inspect startup logs, command,   Artifact exists but runtime contract is wrong.
   container exits         g                env, mounts, and health

   New pods healthy,       Routing/journe   Trace request through            Health check is too shallow or traffic targets wrong version.
   users get errors        y                DNS/CDN/LB/app/dependency

   Rollback code fails     Data             Compare migration and writes     Old release cannot understand current schema/data.
                           compatibility    since cutover

   Only production fails   Environment      Diff config, secrets, flags,     Artifact parity exists but environment contract differs.
                           drift            quotas, regions, and
                                            dependencies

   Redeploy same           Artifact         Compare digests, lockfiles,      Release identity is commit-only and build is not reproducible.
   commit changes          mutability       builder, and base image
   result

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

01 Freeze and identify the release Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: Record pipeline, commit, artifact digest, environment, config/secret/flag versions, migration, rollout, routing, and last-known-good. Pause automatic promotion.

Read the result: There is one exact failed release and one verified recovery target.

Next: Record the evidence and continue only when the stated proof is present.

02 Classify the failing stage Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.

Do: Separate source, dependency, build, test, scan, artifact, deploy control plane, startup, health, routing, database, dependency, and user-journey evidence.

Read the result: The first failing layer is isolated before another deploy.

Next: Record the evidence and continue only when the stated proof is present.

03 Protect data and compatibility Why: Inconsistent inputs create false differences and make later comparisons unreliable.

Do: Inspect migrations, backups, new writes, queues, and old/new version compatibility. Decide rollback versus roll forward based on representational safety.

Read the result: Recovery will not discard accepted data or create incompatible writers.

Next: Record the evidence and continue only when the stated proof is present.

04 Recover with the narrowest action Why: A decisive test reduces trial-and-error and limits unnecessary change.

Do: Stop rollout, shift traffic to known-good compatible release, correct config/secret/route, or deploy a minimal forward fix. Retain failed artifacts and logs.

Read the result: User impact falls and authoritative state remains consistent.

Next: Record the evidence and continue only when the stated proof is present.

Procedure continued 05 Verify layered health Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.

Do: Check platform status, process, readiness, logs, dependency calls, database, auth, critical API, browser journey, observability, and external routing.

Read the result: The requested product outcome works, not just the deployment command.

Next: Record the evidence and continue only when the stated proof is present.

06 Repair pipeline controls Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.

Do: Pin dependencies/base images, build immutable artifacts once, sign/attest where appropriate, validate config schema, enforce migration and canary gates, and retain provenance.

Read the result: The failure class is blocked or detected before broad traffic.

Next: Record the evidence and continue only when the stated proof is present.

07 Rehearse stop and rollback Why: A result is not complete until it remains observable and repeatable after the immediate fix.

Do: Inject failed startup, bad health, config error, dependency outage, migration incompatibility, and partial rollout. Measure detection and recovery.

Read the result: Pipeline stops automatically and operator recovery meets the target.

Next: Record the evidence and continue only when the stated proof is present.

Operational worksheet Evidence record Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.

• Pipeline/run/release IDs, commit/tag, artifact/image digest, builder, and dependency lock.
• Environment config/secret/flag versions and change approvals.
• Migration state, compatibility window, backup/restore proof.
• Build/test/security/deploy logs, rollout events, health, traces, metrics, and user journey.
• Last-known-good release, traffic/routing state, rollback/roll-forward, and incident record.

Acceptance scoreboard

• Release identity includes immutable artifact digest and exact environment configuration versions.
• First failing layer is isolated from build through user journey.
• Migration/data compatibility determines rollback versus roll forward.
• Recovery preserves failed evidence and last-known-good service.
• Layered and end-to-end checks prove actual product behavior.
• Pipeline provenance, config validation, canary, stop, and rollback failure tests pass.

Decision rule SHIP / AUTOMATE GATE Proceed only when every required acceptance check is supported by direct evidence, rollback is available, and the remaining risk is explicitly owned. Unknown is not a pass.

Minimum handoff record

• Versioned ci/cd failed deployment recovery scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Worked example

Starting Problem:

A production deploy is green in the platform but every login returns 500 because the secret name changed.

Evidence collected

• Container readiness checks only /health without dependencies.
• Artifact digest matches staging.
• Production secret uses old key name.
• No configuration schema check exists.

Decision The artifact is valid; the production environment contract and health gate are incomplete. Restore compatible secret mapping or roll forward safely.

Actions taken

• Paused rollout and routed to last-known-good.
• Validated production config against a schema without exposing values.
• Added authenticated dependency-aware canary.
• Promoted the same artifact through staging and production.

Proof Of Completion:

A missing/renamed secret fails preflight before traffic; canary login passes; rollback remains compatible and measured.

Why this example matters The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.

Verify, recover, and hand off

Completion tests A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.

• Release identity includes immutable artifact digest and exact environment configuration versions.
• First failing layer is isolated from build through user journey.
• Migration/data compatibility determines rollback versus roll forward.
• Recovery preserves failed evidence and last-known-good service.
• Layered and end-to-end checks prove actual product behavior.
• Pipeline provenance, config validation, canary, stop, and rollback failure tests pass.

Rollback or safe recovery

• Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
• Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
• Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
• Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.

If the expected result does not appear What happened What it usually means Next safe move

Build passes, container exits Artifact exists but runtime contract is Inspect startup logs, command, env, mounts, and health wrong.

New pods healthy, users get Health check is too shallow or traffic Trace request through DNS/CDN/LB/app/dependency errors targets wrong version.

Rollback code fails Old release cannot understand current Compare migration and writes since cutover schema/data.

Only production fails Artifact parity exists but environment Diff config, secrets, flags, quotas, regions, and dependencies contract differs.

Reusable handoff record

• Versioned ci/cd failed deployment recovery scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Agent delivery contract

    Required inputs
       Field                           Type                  Requirement

       target                          object                Versioned environment, resource, identity, or workflow being evaluated.

       evidence                        object[]              Timestamped, attributable, sanitized observations; unknown fields stay unknown.

       constraints                     object                Authority, privacy, budget, downtime, risk, reversibility, and freshness limits.

       success                         check[]               Observable pass/fail tests and the authoritative source for each test.

    Returned output
       Field                           Type                  Requirement

       diagnosis                       object                Likely layer, supporting and conflicting evidence, alternatives, and confidence.

       plan                            step[]                Ordered bounded actions with owner, risk, expected proof, and stop condition.

       verification                    check[]               Observed pass/fail/unknown results, not inferred success from command exit alone.

       handoff                         object                Sanitized evidence record, recovery state, remaining risk, and next review trigger.

    Agent refusal and escalation rules
•
      Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
•
      Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
•
      Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
•
      Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.

    Confidence rule
    Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low
    confidence; a controlled test that isolates the layer and passes verification can support high confidence.

Official reference starting points

• https://slsa.dev/
• https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions
• https://kubernetes.io/docs/concepts/workloads/controllers/deployment/