SAYLOR This guide turns an Ubuntu error into an evidence-based diagnosis, the smallest safe correction, a verification test, and a rollback path.
The result you're building
A reproducible incident record containing the exact failure, the component that caused it, one narrowly scoped correction, proof that the original task now works, and a rollback path.
Use when: an Ubuntu command, service, package op, device, or login fails and the error doesn't name the real layer; you're helping someone without taking control of their machine; you want a clean record to hand to a technician or agent.
Not a substitute for: a failing storage device, active data loss, suspected compromise, or electrical fault (preserve data, escalate first); blindly applying commands one hypothesis at a time is the point, not a shortcut.
Before you change anything, collect: Ubuntu release/kernel (lsb_release -ds, uname -r); the complete command and complete error text, including the *first* error; whether it started after an update/reboot/install/permission or hardware change; df -h/df -i; relevant service/journal lines with secrets redacted.
Understand the system first
- An error names where detection happened, not always where failure began — work backward from observer to the dependency that supplied the bad state.
- The first failing event is more valuable than the loudest later one — capture the first non-zero exit, first failed unit, first kernel error before cleanup erases it.
- A safe repair changes the smallest layer the evidence supports: restart before reinstall, repair one package before deleting databases, known-good kernel before reinstalling Ubuntu.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | Meaning |
|---|---|---|---|
No space left, write failure | Storage | df -h; df -i; findmnt -no OPTIONS / | Full bytes/inodes or read-only mount must resolve first |
permission denied | Identity/path | namei -l PATH + service user | First path component without traverse/rw identifies boundary |
| Command/library missing | Package/runtime | command -v NAME; dpkg -S PATH; apt-cache policy PKG | Missing package, wrong PATH, mixed repo, or wrong arch |
| Service inactive/failed/restarts | systemd/app | systemctl status UNIT; journalctl -u UNIT -b | Earliest journal error determines branch |
| Freeze, no signal | Kernel/driver/hw | journalctl -k -b -p warning..alert; lspci -nnk | Decides software rollback vs hardware inspection |
Step-by-step procedure
01 Reproduce once, freeze evidence. Run the smallest failing command once; record command, exit code, time, full output.
failing-command
printf 'exit=%s time=%s\n' "$?" "$(date --iso-8601=seconds)"02 Check platform health first. Disk/memory/kernel faults can make unrelated apps fail misleadingly.
df -h; df -i; free -h
systemctl --failed --no-pager
journalctl -k -b -p warning..alert --no-pager | tail -n 8003 Identify the owning component.
command -v COMMAND
dpkg -S /FULL/PATH 2>/dev/null
systemctl cat UNIT
namei -l /FULL/PATHUnexpected /usr/local paths, mixed package owners, and drop-in overrides are strong clues.
04 Run a decisive, read-only test.
sudo -u SERVICEUSER test -r /FULL/PATH; echo $?
curl -v http://127.0.0.1:PORT/health
apt-cache policy PACKAGE05 Make one reversible correction.
sudo cp -a /etc/example.conf /etc/example.conf.before
sudo COMMAND-TO-VALIDATE-CONFIG
sudo systemctl restart UNITFailed validation means restore the backup, not stack another change.
06 Prove the original task and adjacent health.
failing-command
systemctl is-active UNIT
journalctl -u UNIT --since '-5 minutes' --no-pagerWorked example
A web app returns 502 after reboot. nginx answers (DNS/TLS fine); local curl to the upstream port is refused; the app unit failed with status=203/EXEC; systemctl cat shows ExecStart pointing at a venv binary that no longer exists. Decision: the executable path is the first broken layer, not nginx/DNS/TLS. Fix: restored the venv from locked dependencies, validated the exact ExecStart path, restarted only that unit, retested local health before the public hostname. Proof: local and public both return 200, no new journal errors in five minutes.
Verify, recover, hand off
Complete when: the exact original command succeeds; the unit stays active through a normal cycle; no new warning+ errors; capacity/permissions/dependencies in bounds; change and rollback location recorded.
Rollback: restore the backed-up config; use a known-good kernel from GRUB rather than removing the current one while booted into it; if things get less stable, stop and return to last known-good.
| Symptom | Usually means | Next move |
|---|---|---|
| Symptom changes each command | No stable baseline | Stop, restore backups, reproduce one minimal failure |
Works only with sudo | Identity/permission/env differs | Test as real service user; inspect namei, groups, ACLs |
| Active but task fails | Shallow status check | Test the real operation and endpoint route |
| No log event at failure time | Wrong unit/boot/logging disabled | Confirm timestamp, -b, unit name, log destination |
Handoff record: original command/timestamp/exit code/sanitized error; release/kernel/versions; isolating evidence and ruled-out alternatives; exact change + rollback command; verification results and remaining uncertainty.
For agents
Structure programmatic use as: inputs — error (complete text/command, not a screenshot), system (release/kernel/arch/session), evidence (timestamped, secret-redacted), authority (diagnose / propose / execute-reversible — never inferred); output — diagnosis (layer, evidence, alternatives, confidence), plan (ordered steps + risk + expected evidence), verification (pass/fail checks), handoff (sanitized record, risks, rollback state). Refuse requests needing a secret/seed/key/credential in ordinary input; stop when the action exceeds declared authority/budget/reversibility; escalate on missing or stale evidence. Score confidence from independent-observation quality, not familiarity — high confidence requires a decisive isolating test plus a verified repair.
References
- https://help.ubuntu.com/
- https://manpages.ubuntu.com/
- https://documentation.ubuntu.com/release-notes/24.04/