SAYLOR The result you are building
Finished Result:
A reproducible container build and run whose image, command, environment, mounts, user, network, health, and persistent data are understood, with the failure isolated without deleting valuable volumes.
Use this guide when
- A Docker build fails, container exits/restarts, health is unhealthy, networking fails, or files disappear/deny
access.
- A Compose stack works on one host but not another.
Do not use it as a substitute for
- Do not use
docker system prune --volumes, delete volumes, or recreate databases before identifying
data ownership and backups.
- Do not solve permission errors by running every container privileged/root.
Before you change anything
Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Image digest/Dockerfile/Compose config and Docker Engine version.
- Container inspect, exit code, state/health, command, environment names, mounts, and network.
- Application logs and daemon logs at the first failure.
- Volume backup/ownership and expected data persistence.
Stop Before Proceeding:
Stop before any prune, volume removal, migration, or forced recreation if a volume may contain the only copy of data. Back up or snapshot it first.
Understand the system before fixing it
Image, container, and volume are different state Rebuilding an image does not repair a corrupt volume; recreating a container can change ephemeral state; deleting a volume can destroy persistent data.
Container localhost is not host localhost Each network namespace has its own loopback. Services reach one another by network name/alias and published ports expose host access.
Exit code and health probe answer different questions A running process can be unhealthy; a one-shot process can exit zero normally. Interpret behavior against service type.
Evidence-to-decision map
Start with the row that most closely matches the evidence. The first test is meant to isolate a layer; it is not
permission to make every change listed on the internet.
Evidence Likely layer First decisive check What the result means
Build step fails Image build Rebuild target with plain Dependency, context, architecture, DNS, or
progress and no cache only for command failure.
suspect step
Exits immediately Entrypoint/app Inspect state exit code and logs Bad command/config, missing dependency,
OOM, or normal one-shot completion.
Permission denied on UID/GID/label Compare container user with Bind-mount identity or confinement mismatch.
mount host path ownership
Connection refused Network/readin Test listener and service DNS Wrong host/port, bind to loopback, or
between services ess inside network dependency not ready.
Unhealthy but app Healthcheck Run exact probe inside Probe path/tool/timeout/user differs from real
works container service.Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
01 Freeze current data and inspect state Why: Recreate/prune can erase the best evidence or data.
Do: Record image digest, container state/exit/OOM, mounts, networks, health, command, and logs. Back up valuable volumes.
docker inspect CONTAINER docker logs --timestamps CONTAINER docker system df docker volume inspect VOLUME
Read the result: Exit 137/OOMKilled points to memory; exit 126/127 to execution/path; application codes require app logs.
Next: Choose build, start, storage, network, or health branch.
02 Render effective Compose configuration Why: Overrides and environment substitution change what actually runs.
Do: Use docker compose config, inspect resolved image/build, command, env names, mounts, ports, depends, and healthcheck; redact values.
docker compose config --no-interpolate docker compose ps -a
Read the result: Unexpected empty variables or overridden commands explain environment-specific behavior.
Next: Do not publish rendered secrets.
03 Reproduce the smallest failing layer Why: Starting the entire stack hides dependency order and cross-service effects.
Do: Build one target or run one container with the same user/mount/env/network. Execute exact health command inside it.
Read the result: If base container works, add mounts/env/dependencies one at a time.
Next: Preserve successful minimal command.
04 Correct identity, storage, or network narrowly Why: Privileged mode and recursive chown broaden risk.
Do: Match service UID/GID to intended volume ownership, create required directories, use named volumes for persistent data, service names for internal DNS, and correct bind address.
docker exec CONTAINER id docker exec CONTAINER ss -ltnp docker exec CONTAINER getent hosts SERVICE
Read the result: Container user must access only required paths; internal service must listen on reachable interface.
Next: Restart only affected service after backup.
05 Validate health and persistence Why: A page load does not prove restart-safe data.
Do: Test health/core operation, restart container, verify data, resource use, and logs; then recreate container without removing volume.
Read the result: State must survive expected lifecycle and health must reflect real readiness.
Next: Do not test destructive lifecycle on unbacked production data.
06 Make the run reproducible Why: Mutable tags and undocumented env recreate failures.
Do: Pin image digests/versions, keep a non-secret env schema, explicit volumes/networks/health, resource limits, backup/restore, and deploy test.
Read the result: A clean host should reproduce with documented inputs.
Next: Record Docker/Compose versions.
Worked example
Starting Problem:
A database container restarts and the application reports connection refused after moving hosts.
Evidence collected
- Database exits with permission denied on its data directory.
- Bind mount files are owned by a UID from the old host.
- Container runs as a different numeric UID on the new image/host.
- Volume contains production data with no current backup.
Decision Storage identity is the first failure; networking is a downstream symptom.
Actions taken
- Stopped stack and made a verified backup of the data directory.
- Confirmed image-required UID/GID and changed ownership only on the dedicated data path.
- Started database, verified readiness, then application and core query.
Proof Of Completion:
Database stays healthy through restart; app connects by service name; data count/checksum matches backup baseline; no privileged mode used.
Why this example matters Protecting the volume before changing ownership preserved the real asset.
Verify, recover, and hand off
Completion tests A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Build uses pinned inputs and completes reproducibly.
- Container state/exit/health match intended service type.
- Core request works across intended network path.
- Data persists through restart/recreate and backup/restore is documented.
- Service runs without unnecessary root/privileged access.
- Logs and resource limits show stable operation.
Rollback or safe recovery
- Restore Compose/config/image version and volume backup if migration or ownership change fails.
- Recreate only containers/networks after confirming persistent volumes are retained.
- Disable new service and reattach prior named volume/image for recovery.
If the expected result does not appear What happened What it usually means Next safe move
Container disappears Compose cleanup or auto-remove Use ps -a, disable --rm, and capture obscures state. inspect/logs.
Host can reach port; peer cannot Published port differs from internal Use service DNS and container port internally. network path.
Works as root UID/GID/path issue remains. Fix dedicated path ownership/permissions and run intended user.
Rebuild changes behavior Mutable base/tag/dependencies or Pin digest/lockfiles and record platform. architecture.
Reusable handoff record Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Image digest, Docker/Compose versions, and effective config.
- Container inspect/logs and isolated failing layer.
- Volume inventory, backup, ownership, and persistence test.
- Network/listener/health/core-operation results.
- Exact fix, rollback, and reproducible start procedure.
Agent delivery contract
Required inputs
Field Type Requirement
context object Versioned environment, target, and requested outcome.
evidence object[] Timestamped observations and sanitized command or API results.
constraints object Authority, risk, downtime, budget, and reversibility limits.
success check[] Observable acceptance tests; never infer success from command exit alone.
Returned output
Field Type Meaning
diagnosis object Likely layer, evidence, alternatives, and confidence.
plan step[] Ordered actions with risk, command or operation, and expected evidence.
verification check[] Pass/fail checks that prove the requested outcome.
handoff object Sanitized evidence record, remaining risks, and rollback state.
Agent refusal and escalation rules
• Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
• Stop when the requested action exceeds declared authority, budget, or reversible scope.
• Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule
Score confidence from the number and quality of independent observations, not from how familiar the
error looks. Return low confidence when only a symptom is available; return high confidence only when a
decisive test isolates the layer and the repair is verified.Official reference starting points
- https://docs.docker.com/engine/daemon/troubleshoot/
- https://docs.docker.com/reference/cli/docker/container/inspect/
- https://docs.docker.com/engine/storage/volumes/