SAYLOR Recover from capacity, inode, log, mount, filesystem, and boot failures while protecting irreplaceable data and avoiding destructive cleanup.
The result you're building
A bootable system with confirmed filesystem health, enough free bytes and inodes for normal operation, the true space consumer identified, essential data preserved, and a prevention/rollback record.
Use this guide when
- Ubuntu fails to boot/login/update because storage or inodes are exhausted.
- A filesystem remounts read-only, services fail to write, or logs/journals grow unexpectedly.
Do not use it as a substitute for
- Do not delete unknown files, active database files, package databases, or filesystem metadata to gain space.
- Do not run filesystem repair on a mounted read-write filesystem.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Backups and irreplaceable-data locations.
df -h,df -i,findmnt, block devices, and filesystem type.- Kernel/storage errors and boot failure messages.
- Largest top-level directories on the same filesystem and deleted-open files.
Understand the system before fixing it
Full bytes and full inodes are different
Large files consume bytes; millions of tiny files can exhaust inodes with apparent free GB. The remedy must match the resource.
du and df can disagree
Deleted files held open still count in df but not pathname totals. Snapshots, reserved blocks, mounts, and sparse files also matter.
Boot repair follows filesystem health
Repeated writes and package repairs on a failing/read-only filesystem can worsen damage. Verify storage before higher layers.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| 100% blocks, inodes available | Capacity | Same-filesystem du plus lsof +L1 | Large path, deleted-open file, snapshot, or reserved space. |
| Inodes 100% | File count | Count files by top-level path | Cache/session/mail/spool or runaway tiny-file creation. |
| Read-only with I/O errors | Filesystem/device | findmnt and kernel log/SMART | Stop writes; backup and offline fs/device recovery. |
| GRUB/initramfs/emergency | Boot/mount | Identify failed mount/unit and filesystem UUID | Wrong fstab, missing device, full root, or filesystem error. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Choose preservation or cleanup mode
Why: A failing device changes every next action.
Do: Inspect mount options, kernel errors, device health, and backup status from TTY/recovery/live media.
Read the result: Read-only/I/O/SMART faults require preservation; clean full filesystem allows bounded cleanup.
Next: Do not remount read-write just to continue.
Step 02 — Measure bytes and inodes per filesystem
Why: Cleaning another mount does not free root.
Do: Record df, inodes, mounts, and block devices; identify exact full filesystem.
df -hT
df -i
findmnt
lsblk -fRead the result: Choose byte or inode branch.
Next: Keep commands on same filesystem.
Step 03 — Locate consumer without crossing mounts
Why: Broad scans can be slow and misleading.
Do: Use du -x at top-level, then descend only into large directories; inspect journal/package caches, containers, user data, and deleted-open files.
sudo du -xhd1 / 2>/dev/null | sort -h
sudo lsof +L1Read the result: lsof +L1 large deleted file requires service restart/rotation, not pathname deletion.
Next: Protect databases and user data.
Step 04 — Free a small safe reserve first
Why: Package/service tools need working space.
Do: Remove only identified expendable caches or rotate/vacuum logs within retention policy; move irreplaceable data to verified storage rather than delete.
Read the result: Recheck df/inodes after each action.
Next: Aim for operational headroom, not a temporary few MB.
Step 05 — Repair affected services/packages after space
Why: Full disks leave interrupted writes and failed units.
Do: Check filesystem, package audit, failed units, databases, and application integrity after safe capacity exists.
Read the result: The original boot/service failure must disappear without new errors.
Next: Offline fsck only when unmounted and appropriate for filesystem.
Step 06 — Prevent recurrence
Why: Space incidents recur without limits and alerts.
Do: Set log rotation/journal limits, container cleanup policy, retention, quotas, capacity/inode alerts, and backup verification.
Read the result: Simulate alert threshold and document safe cleanup owner.
Next: Keep emergency free-space reserve if appropriate.
Worked example
du totals are far below df, and services cannot start.Evidence collected
- Inodes are available.
lsof +L1shows a 40 GB deleted log still held by one service.- Filesystem has no I/O errors and is read-write.
- The original logfile pathname is already deleted.
Decision: Space is retained by an open file descriptor; deleting more pathnames will not free those blocks.
Actions taken
- Preserved service logs/status, then restarted only the owning service during an authorized window.
- Verified descriptor closed and space returned.
- Configured rotation/reopen behavior and alert.
df returns healthy headroom, service starts, no deleted large descriptors remain, and rotation test reopens log correctly.Why this example matters: Comparing du with df exposed the invisible consumer and avoided random deletion.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Filesystem/device show no unresolved errors before normal writes resume.
- Free bytes and inodes exceed defined operating reserve.
- Largest consumer and cause are identified, not guessed.
- Package/database/services pass integrity and original boot/task test.
- Backups and retention/alerts are verified.
Rollback or safe recovery
- Restore moved/deleted data from verified backup or trash when cleanup was mistaken.
- Revert retention/rotation settings if they remove required records.
- Boot live/recovery media and restore filesystem/image when health degrades.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Space returns briefly then fills | Active producer continues. | Measure growth over time and identify PID/path; stop cause. |
du much smaller than df | Deleted-open file, snapshot, reserved blocks, or hidden mount. | Check lsof, snapshots, mounts, filesystem tools. |
| Inodes full | Too many files, not large bytes. | Count by directory and remove through owning application policy. |
| Read-only remount | Filesystem protected itself after error. | Stop cleanup; preserve data and inspect/offline repair. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Filesystem/device/backup health decision.
- Byte/inode/mount measurements and consumer evidence.
- Exact safe cleanup with before/after capacity.
- Post-full package/service/data integrity results.
- Retention, alert, backup, and recovery policy.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points