SAYLOR Turn a fresh Ubuntu server into a documented, patched, least-privileged, observable, recoverable service host without locking out the operator.
The result you're building
A reproducible Ubuntu 24.04 server baseline with verified administrative access, patch policy, firewall, service isolation, time/logging, monitoring, backups, recovery access, and an evidence-backed hardening report.
Use this guide when
- You are preparing a new VPS for a website, API, agent, database, or client service.
- An inherited server lacks a clear security and recovery baseline.
- You need repeatable checks before exposing a service publicly.
Do not use it as a substitute for
- Running a one-line hardening script without understanding provider recovery.
- Disabling access, changing SSH, or enforcing firewall rules before a second verified session and console path exist.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Provider, region, image, Ubuntu release/kernel, IPs, DNS, and ownership.
- Console/rescue access, authorized administrators, SSH keys, and recovery contacts.
- Listening sockets, processes, packages, services, users, groups, sudo, and scheduled jobs.
- Firewall/security-group rules, routes, DNS, NTP, logging, and update policy.
- Backup scope, snapshot policy, restore proof, service health, and rollback checkpoint.
Understand the system before fixing it
Observe before mutating
Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.
Recovery must be exercised
A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.
Provider controls and guest controls both matter
Cloud firewall, console, snapshots, metadata, IAM, and abuse response sit outside Ubuntu; document both layers.
Hardening must preserve operation
A secure server that cannot be recovered, patched, monitored, or administered safely is not production-ready.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Public port open despite UFW deny | Cloud/network path | Compare provider firewall, nftables, sockets, and external probe | Traffic may bypass expected layer or service listens on another interface/port. |
| SSH key rejected after hardening | SSH/auth | Use console to inspect effective sshd config and logs | Include order, permissions, user, key, or algorithm policy is wrong. |
| Updates installed but reboot pending | Kernel/runtime | Compare running kernel and reboot-required state | Patched packages are not active until controlled restart. |
| Disk grows unexpectedly | Logs/data | Inspect journal, application logs, cache, and inode use | Retention or runaway service can make the host unavailable. |
| Snapshot exists but app restore fails | Recovery | Restore to isolated VPS and run service checks | Snapshot omits external data, secrets, DNS, or runbook details. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Record ownership and recovery path
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Document provider account, billing owner, region, instance/image, IP/DNS, console/rescue access, recovery contacts, and authorized administrators. Enable strong account protection.
Read the result: A verified owner can reach console without relying on the server's SSH path.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Capture the clean baseline
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Record release/kernel, hardware, disks, mounts, network, routes, DNS, time, packages, repos, services, sockets, users, sudo, jobs, and logs. Take a labeled snapshot if policy permits.
Read the result: Before-state and rollback point are complete and attributable.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Establish least-privileged administration
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Create named admin accounts with SSH keys and scoped sudo; verify from a second session. Disable or restrict password/root login only after effective-config tests.
Read the result: Two independent authorized access paths work; unauthorized password/root tests fail.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Patch and control software sources
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Use official or justified repositories, update packages, review removals, set unattended security policy, schedule reboots, and record exceptions.
Read the result: Security updates apply without broken dependencies and running kernel state is known.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Restrict network and services
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Remove/disable unused services, bind internal listeners appropriately, configure provider firewall plus nftables/UFW from required-flow matrix, and rate-limit exposed authentication.
Read the result: External probes show only documented ports and required internal flows still work.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Add isolation, logging, and monitoring
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Use dedicated service users, restrictive files, systemd protections where compatible, NTP, persistent/remote logs as needed, health checks, resource alerts, and audit coverage.
Read the result: Service failure, auth attempts, resource pressure, and clock state are observable.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Prove backup, recovery, and handoff
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Back up data/config/secrets references, restore into isolation, test application journeys and console recovery, then document update, restart, rollback, and incident steps.
Read the result: A clean operator restores the minimum viable service within declared targets.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Provider, region, image, Ubuntu release/kernel, IPs, DNS, and ownership.
- Console/rescue access, authorized administrators, SSH keys, and recovery contacts.
- Listening sockets, processes, packages, services, users, groups, sudo, and scheduled jobs.
- Firewall/security-group rules, routes, DNS, NTP, logging, and update policy.
- Backup scope, snapshot policy, restore proof, service health, and rollback checkpoint.
Acceptance scoreboard
- Provider ownership, console, billing, IP/DNS, and recovery contacts are documented.
- Release, packages, services, users, jobs, sockets, firewall, and time baseline is preserved.
- Named key-based admin and independent recovery access are proven before restrictions.
- Security updates, reboot policy, and repository trust are controlled.
- External probes expose only required ports; service users and files are least-privileged.
- Monitoring, backups, isolated restore, rollback, and operator runbook pass.
Minimum handoff record
- Versioned ubuntu vps production baseline scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- No provider console test was performed.
- The original root session was the only access path.
- Effective sshd config was not checked.
- No snapshot or rollback record exists.
Decision: The sequence was unsafe. Use provider rescue/console, restore authorized access, then repeat changes one at a time with a second session.
Actions taken
- Recovered through provider console.
- Verified key ownership and sshd effective config.
- Tested new admin in parallel session.
- Applied firewall and auth changes separately with external probes.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Provider ownership, console, billing, IP/DNS, and recovery contacts are documented.
- Release, packages, services, users, jobs, sockets, firewall, and time baseline is preserved.
- Named key-based admin and independent recovery access are proven before restrictions.
- Security updates, reboot policy, and repository trust are controlled.
- External probes expose only required ports; service users and files are least-privileged.
- Monitoring, backups, isolated restore, rollback, and operator runbook pass.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Public port open despite UFW deny | Traffic may bypass expected layer or service listens on another interface/port. | Compare provider firewall, nftables, sockets, and external probe |
| SSH key rejected after hardening | Include order, permissions, user, key, or algorithm policy is wrong. | Use console to inspect effective sshd config and logs |
| Updates installed but reboot pending | Patched packages are not active until controlled restart. | Compare running kernel and reboot-required state |
| Disk grows unexpectedly | Retention or runaway service can make the host unavailable. | Inspect journal, application logs, cache, and inode use |
Reusable handoff record
- Versioned ubuntu vps production baseline scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points