systemd Failure Resolver

Diagnose a failed systemd unit from its real execution context, dependencies, permissions, environment, restart policy, and journal.

The result you're building

A systemd unit that starts reliably under its real service identity and environment, passes application readiness, obeys dependency/restart limits, and has a documented failure cause and rollback.

Use this guide when

A service is failed, restart-looping, works manually but not under systemd, or starts before a dependency.
A daemon loses environment variables, permissions, paths, or network access after reboot.

Do not use it as a substitute for

Do not paste shell syntax into ExecStart unless explicitly invoking a shell with safe quoting.
Do not use restart loops to hide a deterministic startup failure.

Before you change anything

Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
Unit name and full systemctl cat output.
Status, exit code, journal for current boot, and first error.
Intended service user/group, working directory, files, ports, and environment.
Dependency/readiness expectations and recent unit/drop-in changes.

Stop before proceeding: Stop before weakening sandboxing, changing ownership recursively, or running the service as root merely to make it start. Isolate the denied resource and grant only required access.

Understand the system before fixing it

systemd does not run your interactive shell
PATH, working directory, environment files, umask, limits, and credentials differ. Absolute paths and explicit configuration make the service reproducible.

Active is not ready
A process may be running before its socket, database, migration, or core operation is usable. Readiness must test the service contract.

Exit status identifies the boundary
203/EXEC, 200/CHDIR, signal exits, timeout, and application codes lead to different tests.

Evidence-to-decision map

Evidence	Likely layer	First decisive check	What the result means
203/EXEC	Executable	`systemctl show -p ExecStart UNIT` and `namei -l`	Missing/non-executable path, wrong interpreter, filesystem mount, or permission.
200/CHDIR	WorkingDirectory	`systemctl show -p WorkingDirectory UNIT`	Directory absent or inaccessible to service user.
Permission denied	Identity/sandbox	Run read-only access test as service user; inspect unit hardening	Unix mode/ACL, groups, SELinux/AppArmor, or systemd restriction.
Restart loop	Application/config	Earliest journal error and `Restart=` policy	Fix deterministic cause; rate limit is secondary.
Manual works	Environment/context	`systemctl show` vs shell env/path/limits	Make dependency explicit in unit/config.

Step-by-step procedure

Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.

Step 01 — Capture effective unit and first failure

Why: Drop-ins can override the file you are reading.

Do: Save systemctl cat, show properties, status, and current-boot journal before editing.

systemctl cat UNIT
systemctl show UNIT -p User -p Group -p ExecStart -p WorkingDirectory -p Environment
systemctl status UNIT --no-pager -l
journalctl -u UNIT -b --no-pager

Read the result: Use the earliest error and exit status, not repeated restart messages.

Next: Choose executable, directory, permission, dependency, or application branch.

Step 02 — Reproduce under service identity

Why: Interactive success under your user does not match systemd context.

Do: Test executable existence, config read, directory traversal, port/path access as the configured service user without starting a duplicate daemon.

sudo -u SERVICEUSER test -x /ABSOLUTE/EXECUTABLE; echo $?
sudo -u SERVICEUSER test -r /PATH/CONFIG; echo $?
namei -l /ABSOLUTE/EXECUTABLE

Read the result: A failed test/read isolates permissions or path; successful access shifts to environment/application.

Next: Never copy secrets into command history.

Step 03 — Validate application outside restart loop

Why: systemd can obscure fast exit output.

Do: Use the application's config-check or foreground/test command as the service user with explicit env file.

Read the result: Fix the first app validation error before restarting the unit.

Next: Keep production port/state safe while testing.

Step 04 — Correct unit with a drop-in

Why: Vendor units should remain package-managed.

Do: Use systemctl edit UNIT for the narrow override; absolute paths, explicit working directory/environment file, dependencies, and restart policy.

sudo systemctl edit UNIT
systemd-analyze verify /etc/systemd/system/UNIT.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart UNIT

Read the result: systemd-analyze verify and daemon reload must be clean.

Next: Restart once and inspect fresh logs.

Step 05 — Test readiness and dependency behavior

Why: Startup success alone can race downstream consumers.

Do: Call local health/core operation, verify listener and dependency ordering, then reboot or restart dependencies if authorized.

systemctl is-active UNIT
ss -ltnp
curl -fsS http://127.0.0.1:PORT/health

Read the result: Ready behavior should be stable without rapid restarts.

Next: Tune timeouts/restart only after root cause is fixed.

Step 06 — Document recovery

Why: Future operators need effective configuration and a known-good path.

Do: Save drop-in, version, service identity, config locations, health test, journal query, and rollback command.

Read the result: A clean reboot must reproduce success.

Next: Back up the drop-in before later edits.

Worked example

Starting problem: A Python service works from the shell but systemd exits 203/EXEC.

Evidence collected

Effective ExecStart points to /home/user/app/venv/bin/python.
The service runs as appsvc, which cannot traverse /home/user.
The project was moved but the unit path was not updated.
Application config validation succeeds from the new service-owned path.

Decision: The executable path/context is invalid; running as root would mask rather than solve it.

Actions taken

Placed release and venv under a service-owned application directory.
Updated only the drop-in ExecStart/WorkingDirectory.
Verified unit, restarted, tested health, and rebooted.

Proof of completion: Unit stays active, health/core request pass, service runs as appsvc, and no permission/exec errors return after reboot.

Why this example matters: Reproducing under the service identity exposed the real boundary.

Verify, recover, and hand off

Completion tests

A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
Effective unit matches intended user, paths, environment, and dependencies.
Unit starts without restart loop and stays within limits.
Local readiness and one core operation pass.
Logs contain no new startup error.
Reboot/dependency restart behavior is documented and tested.

Rollback or safe recovery

Remove/restore the drop-in, daemon-reload, and restart the prior configuration.
Use systemctl revert UNIT only after saving current override and understanding vendor state.
Switch to multi-user/recovery access if the failed unit blocks graphical/remote login.

If the expected result does not appear

What happened	What it usually means	Next safe move
Edit has no effect	Different drop-in precedence or daemon not reloaded.	Inspect `systemctl cat` and reload; edit effective source.
Status active; port absent	Process is not ready/listens elsewhere.	Inspect app logs/config and actual sockets.
Permission works manually as service user	systemd sandbox or mount namespace differs.	Inspect `systemctl show` hardening properties.
Starts only after manual delay	Dependency/order/readiness is missing.	Use correct `After`/`Requires` and application retry/readiness, not sleep if avoidable.

Reusable handoff record

Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
Effective unit/drop-ins and pre-fix journal.
Exit-status interpretation and decisive identity/path test.
Narrow override/config change and validation.
Readiness/core-operation and reboot results.
Rollback and support commands.

Agent delivery contract

Commercial boundary: Human-readable use remains free. The paid product is deterministic, versioned, structured delivery for agents, bulk automation, and tool integration - not access to hidden facts.

Required inputs

Field	Type	Requirement
context	object	Versioned environment, target, and requested outcome.
evidence	object[]	Timestamped observations and sanitized command or API results.
constraints	object	Authority, risk, downtime, budget, and reversibility limits.
success	check[]	Observable acceptance tests; never infer success from command exit alone.

Returned output

Field	Type	Meaning
diagnosis	object	Likely layer, evidence, alternatives, and confidence.
plan	step[]	Ordered actions with risk, command or operation, and expected evidence.
verification	check[]	Pass/fail checks that prove the requested outcome.
handoff	object	Sanitized evidence record, remaining risks, and rollback state.

Agent refusal and escalation rules

Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
Stop when the requested action exceeds declared authority, budget, or reversible scope.
Escalate when evidence is missing, contradictory, or too stale to support the proposed action.

Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.

Official reference starting points

What are you seeing?

The result you're building

Use this guide when

Do not use it as a substitute for

Before you change anything

Understand the system before fixing it

Step-by-step procedure

Step 01 — Capture effective unit and first failure

Step 02 — Reproduce under service identity

Step 03 — Validate application outside restart loop

Step 04 — Correct unit with a drop-in

Step 05 — Test readiness and dependency behavior

Step 06 — Document recovery

Worked example

Verify, recover, and hand off

Agent delivery contract

Related guides