The result you are building

Finished Result:

A bootable system with confirmed filesystem health, enough free bytes and inodes for normal operation, the true space consumer identified, essential data preserved, and a prevention/rollback record.

Use this guide when

Ubuntu fails to boot/login/update because storage or inodes are exhausted.
A filesystem remounts read-only, services fail to write, or logs/journals grow unexpectedly.

Do not use it as a substitute for

Do not delete unknown files, active database files, package databases, or filesystem metadata to gain

space.

Do not run filesystem repair on a mounted read-write filesystem.

Before you change anything

Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.

Backups and irreplaceable-data locations.
df -h, df -i, findmnt, block devices, and filesystem type.
Kernel/storage errors and boot failure messages.
Largest top-level directories on the same filesystem and deleted-open files.

Stop Before Proceeding:

Stop normal cleanup if the filesystem is read-only, reports I/O/metadata errors, SMART/device health is bad, or important data has no backup. Switch to preservation/recovery media and minimize writes.

Understand the system before fixing it

Full bytes and full inodes are different Large files consume bytes; millions of tiny files can exhaust inodes with apparent free GB. The remedy must match the resource.

du and df can disagree Deleted files held open still count in df but not pathname totals. Snapshots, reserved blocks, mounts, and sparse files also matter.

Boot repair follows filesystem health Repeated writes and package repairs on a failing/read-only filesystem can worsen damage. Verify storage before higher layers.

Evidence-to-decision map

  Start with the row that most closely matches the evidence. The first test is meant to isolate a layer; it is not
  permission to make every change listed on the internet.

    Evidence                  Likely layer       First decisive check              What the result means

    100% blocks, inodes       Capacity           Same-filesystem `du` plus `lsof   Large path, deleted-open file, snapshot, or
    available                                    +L1`                              reserved space.

    Inodes 100%               File count         Count files by top-level path     Cache/session/mail/spool or runaway tiny-file
                                                                                   creation.

    Read-only with I/O        Filesystem/devi    `findmnt` and kernel              Stop writes; backup and offline fs/device
    errors                    ce                 log/SMART                         recovery.

    GRUB/initramfs/emer       Boot/mount         Identify failed mount/unit and    Wrong fstab, missing device, full root, or
    gency                                        filesystem UUID                   filesystem error.

Step-by-step procedure

Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.

01 Choose preservation or cleanup mode Why: A failing device changes every next action.

Do: Inspect mount options, kernel errors, device health, and backup status from TTY/recovery/live media.

Read the result: Read-only/I/O/SMART faults require preservation; clean full filesystem allows bounded cleanup.

Next: Do not remount read-write just to continue.

02 Measure bytes and inodes per filesystem Why: Cleaning another mount does not free root.

Do: Record df, inodes, mounts, and block devices; identify exact full filesystem.

df -hT df -i findmnt lsblk -f

Read the result: Choose byte or inode branch.

Next: Keep commands on same filesystem.

03 Locate consumer without crossing mounts Why: Broad scans can be slow and misleading.

Do: Use du -x at top-level, then descend only into large directories; inspect journal/package caches, containers, user data, and deleted-open files.

sudo du -xhd1 / 2>/dev/null | sort -h sudo lsof +L1

Read the result: lsof +L1 large deleted file requires service restart/rotation, not pathname deletion.

Next: Protect databases and user data.

04 Free a small safe reserve first Why: Package/service tools need working space.

Do: Remove only identified expendable caches or rotate/vacuum logs within retention policy; move irreplaceable data to verified storage rather than delete.

Read the result: Recheck df/inodes after each action.

Next: Aim for operational headroom, not a temporary few MB.

05 Repair affected services/packages after space Why: Full disks leave interrupted writes and failed units.

Do: Check filesystem, package audit, failed units, databases, and application integrity after safe capacity exists.

Read the result: The original boot/service failure must disappear without new errors.

Next: Offline fsck only when unmounted and appropriate for filesystem.

06 Prevent recurrence Why: Space incidents recur without limits and alerts.

Do: Set log rotation/journal limits, container cleanup policy, retention, quotas, capacity/inode alerts, and backup verification.

Read the result: Simulate alert threshold and document safe cleanup owner.

Next: Keep emergency free-space reserve if appropriate.

Worked example

Starting Problem:

Root is 100%; du totals are far below df, and services cannot start.

Evidence collected

Inodes are available.
lsof +L1 shows a 40 GB deleted log still held by one service.
Filesystem has no I/O errors and is read-write.
The original logfile pathname is already deleted.

Decision Space is retained by an open file descriptor; deleting more pathnames will not free those blocks.

Actions taken

Preserved service logs/status, then restarted only the owning service during an authorized window.
Verified descriptor closed and space returned.
Configured rotation/reopen behavior and alert.

Proof Of Completion:

df returns healthy headroom, service starts, no deleted large descriptors remain, and rotation test reopens log correctly.

Why this example matters Comparing du with df exposed the invisible consumer and avoided random deletion.

Verify, recover, and hand off

Completion tests A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.

Filesystem/device show no unresolved errors before normal writes resume.
Free bytes and inodes exceed defined operating reserve.
Largest consumer and cause are identified, not guessed.
Package/database/services pass integrity and original boot/task test.
Backups and retention/alerts are verified.

Rollback or safe recovery

Restore moved/deleted data from verified backup or trash when cleanup was mistaken.
Revert retention/rotation settings if they remove required records.
Boot live/recovery media and restore filesystem/image when health degrades.

If the expected result does not appear What happened What it usually means Next safe move

Space returns briefly then fills Active producer continues. Measure growth over time and identify PID/path; stop cause.

du much smaller than df Deleted-open file, snapshot, reserved Check lsof, snapshots, mounts, filesystem blocks, or hidden mount. tools.

Inodes full Too many files, not large bytes. Count by directory and remove through owning application policy.

Read-only remount Filesystem protected itself after error. Stop cleanup; preserve data and inspect/offline repair.

Reusable handoff record Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.

Filesystem/device/backup health decision.
Byte/inode/mount measurements and consumer evidence.
Exact safe cleanup with before/after capacity.
Post-full package/service/data integrity results.
Retention, alert, backup, and recovery policy.

Agent delivery contract

  Required inputs
    Field                        Type               Requirement

    context                      object             Versioned environment, target, and requested outcome.

    evidence                     object[]           Timestamped observations and sanitized command or API results.

    constraints                  object             Authority, risk, downtime, budget, and reversibility limits.

    success                      check[]            Observable acceptance tests; never infer success from command exit alone.

  Returned output
    Field                        Type               Meaning

    diagnosis                    object             Likely layer, evidence, alternatives, and confidence.

    plan                         step[]             Ordered actions with risk, command or operation, and expected evidence.

    verification                 check[]            Pass/fail checks that prove the requested outcome.

    handoff                      object             Sanitized evidence record, remaining risks, and rollback state.

  Agent refusal and escalation rules
  • Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
  • Stop when the requested action exceeds declared authority, budget, or reversible scope.
  • Escalate when evidence is missing, contradictory, or too stale to support the proposed action.

  Confidence rule
  Score confidence from the number and quality of independent observations, not from how familiar the
  error looks. Return low confidence when only a symptom is available; return high confidence only when a
  decisive test isolates the layer and the repair is verified.

Official reference starting points

https://man7.org/linux/man-pages/man1/df.1.html
https://man7.org/linux/man-pages/man1/du.1.html
https://man7.org/linux/man-pages/man8/fsck.8.html

Disk-Full and Boot Recovery Guide