Diagnose build, startup, health, storage, permission, networking, and environment failures without deleting working images or volumes.
The result you're building
A reproducible container build and run whose image, command, environment, mounts, user, network, health, and persistent data are understood, with the failure isolated without deleting valuable volumes.
Use this guide when
- A Docker build fails, container exits/restarts, health is unhealthy, networking fails, or files disappear/deny access.
- A Compose stack works on one host but not another.
Do not use it as a substitute for
- Do not use
docker system prune --volumes, delete volumes, or recreate databases before identifying data ownership and backups. - Do not solve permission errors by running every container privileged/root.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Image digest/Dockerfile/Compose config and Docker Engine version.
- Container inspect, exit code, state/health, command, environment names, mounts, and network.
- Application logs and daemon logs at the first failure.
- Volume backup/ownership and expected data persistence.
Understand the system before fixing it
Image, container, and volume are different state
Rebuilding an image does not repair a corrupt volume; recreating a container can change ephemeral state; deleting a volume can destroy persistent data.
Container localhost is not host localhost
Each network namespace has its own loopback. Services reach one another by network name/alias and published ports expose host access.
Exit code and health probe answer different questions
A running process can be unhealthy; a one-shot process can exit zero normally. Interpret behavior against service type.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Build step fails | Image build | Rebuild target with plain progress and no cache only for suspect step | Dependency, context, architecture, DNS, or command failure. |
| Exits immediately | Entrypoint/app | Inspect state exit code and logs | Bad command/config, missing dependency, OOM, or normal one-shot completion. |
| Permission denied on mount | UID/GID/label | Compare container user with host path ownership | Bind-mount identity or confinement mismatch. |
| Connection refused between services | Network/readiness | Test listener and service DNS inside network | Wrong host/port, bind to loopback, or dependency not ready. |
| Unhealthy but app works | Healthcheck | Run exact probe inside container | Probe path/tool/timeout/user differs from real service. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Freeze current data and inspect state
Why: Recreate/prune can erase the best evidence or data.
Do: Record image digest, container state/exit/OOM, mounts, networks, health, command, and logs. Back up valuable volumes.
docker inspect CONTAINER
docker logs --timestamps CONTAINER
docker system df
docker volume inspect VOLUMERead the result: Exit 137/OOMKilled points to memory; exit 126/127 to execution/path; application codes require app logs.
Next: Choose build, start, storage, network, or health branch.
Step 02 — Render effective Compose configuration
Why: Overrides and environment substitution change what actually runs.
Do: Use docker compose config, inspect resolved image/build, command, env names, mounts, ports, depends, and healthcheck; redact values.
docker compose config --no-interpolate
docker compose ps -aRead the result: Unexpected empty variables or overridden commands explain environment-specific behavior.
Next: Do not publish rendered secrets.
Step 03 — Reproduce the smallest failing layer
Why: Starting the entire stack hides dependency order and cross-service effects.
Do: Build one target or run one container with the same user/mount/env/network. Execute exact health command inside it.
Read the result: If base container works, add mounts/env/dependencies one at a time.
Next: Preserve successful minimal command.
Step 04 — Correct identity, storage, or network narrowly
Why: Privileged mode and recursive chown broaden risk.
Do: Match service UID/GID to intended volume ownership, create required directories, use named volumes for persistent data, service names for internal DNS, and correct bind address.
docker exec CONTAINER id
docker exec CONTAINER ss -ltnp
docker exec CONTAINER getent hosts SERVICERead the result: Container user must access only required paths; internal service must listen on reachable interface.
Next: Restart only affected service after backup.
Step 05 — Validate health and persistence
Why: A page load does not prove restart-safe data.
Do: Test health/core operation, restart container, verify data, resource use, and logs; then recreate container without removing volume.
Read the result: State must survive expected lifecycle and health must reflect real readiness.
Next: Do not test destructive lifecycle on unbacked production data.
Step 06 — Make the run reproducible
Why: Mutable tags and undocumented env recreate failures.
Do: Pin image digests/versions, keep a non-secret env schema, explicit volumes/networks/health, resource limits, backup/restore, and deploy test.
Read the result: A clean host should reproduce with documented inputs.
Next: Record Docker/Compose versions.
Worked example
Evidence collected
- Database exits with permission denied on its data directory.
- Bind mount files are owned by a UID from the old host.
- Container runs as a different numeric UID on the new image/host.
- Volume contains production data with no current backup.
Decision: Storage identity is the first failure; networking is a downstream symptom.
Actions taken
- Stopped stack and made a verified backup of the data directory.
- Confirmed image-required UID/GID and changed ownership only on the dedicated data path.
- Started database, verified readiness, then application and core query.
Why this example matters: Protecting the volume before changing ownership preserved the real asset.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Build uses pinned inputs and completes reproducibly.
- Container state/exit/health match intended service type.
- Core request works across intended network path.
- Data persists through restart/recreate and backup/restore is documented.
- Service runs without unnecessary root/privileged access.
- Logs and resource limits show stable operation.
Rollback or safe recovery
- Restore Compose/config/image version and volume backup if migration or ownership change fails.
- Recreate only containers/networks after confirming persistent volumes are retained.
- Disable new service and reattach prior named volume/image for recovery.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Container disappears | Compose cleanup or auto-remove obscures state. | Use ps -a, disable --rm, and capture inspect/logs. |
| Host can reach port; peer cannot | Published port differs from internal network path. | Use service DNS and container port internally. |
| Works as root | UID/GID/path issue remains. | Fix dedicated path ownership/permissions and run intended user. |
| Rebuild changes behavior | Mutable base/tag/dependencies or architecture. | Pin digest/lockfiles and record platform. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Image digest, Docker/Compose versions, and effective config.
- Container inspect/logs and isolated failing layer.
- Volume inventory, backup, ownership, and persistence test.
- Network/listener/health/core-operation results.
- Exact fix, rollback, and reproducible start procedure.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points