SAYLOR The result you are building
Finished Result:
A reproducible Ubuntu 24.04 server baseline with verified administrative access, patch policy, firewall, service isolation, time/logging, monitoring, backups, recovery access, and an evidence-backed hardening report.
Use this guide when
- You are preparing a new VPS for a website, API, agent, database, or client service.
- An inherited server lacks a clear security and recovery baseline.
- You need repeatable checks before exposing a service publicly.
Do not use it as a substitute for
- Running a one-line hardening script without understanding provider recovery.
- Disabling access, changing SSH, or enforcing firewall rules before a second verified session and console path exist.
Before you change anything
Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Provider, region, image, Ubuntu release/kernel, IPs, DNS, and ownership.
- Console/rescue access, authorized administrators, SSH keys, and recovery contacts.
- Listening sockets, processes, packages, services, users, groups, sudo, and scheduled jobs.
- Firewall/security-group rules, routes, DNS, NTP, logging, and update policy.
- Backup scope, snapshot policy, restore proof, service health, and rollback checkpoint.
Stop Before Proceeding:
Do not change SSH, sudo, networking, encryption, boot, or firewall until provider console/rescue access and a second privileged session are verified. Preserve a snapshot and current configuration first.
Understand the system before fixing it
Observe before mutating Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.
Recovery must be exercised A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.
Provider controls and guest controls both matter Cloud firewall, console, snapshots, metadata, IAM, and abuse response sit outside Ubuntu; document both layers.
Hardening must preserve operation A secure server that cannot be recovered, patched, monitored, or administered safely is not production-ready.
Evidence-to-decision map
Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.
Evidence Likely layer First decisive check What the result means
Public port open Cloud/network Compare provider firewall, Traffic may bypass expected layer or service listens on another
despite UFW deny path nftables, sockets, and external interface/port.
probe
SSH key rejected SSH/auth Use console to inspect effective Include order, permissions, user, key, or algorithm policy is
after hardening sshd config and logs wrong.
Updates installed but Kernel/runtime Compare running kernel and Patched packages are not active until controlled restart.
reboot pending reboot-required state
Disk grows Logs/data Inspect journal, application Retention or runaway service can make the host unavailable.
unexpectedly logs, cache, and inode use
Snapshot exists but Recovery Restore to isolated VPS and run Snapshot omits external data, secrets, DNS, or runbook details.
app restore fails service checksStep-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
01 Record ownership and recovery path Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Document provider account, billing owner, region, instance/image, IP/DNS, console/rescue access, recovery contacts, and authorized administrators. Enable strong account protection.
Read the result: A verified owner can reach console without relying on the server's SSH path.
Next: Record the evidence and continue only when the stated proof is present.
02 Capture the clean baseline Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Record release/kernel, hardware, disks, mounts, network, routes, DNS, time, packages, repos, services, sockets, users, sudo, jobs, and logs. Take a labeled snapshot if policy permits.
Read the result: Before-state and rollback point are complete and attributable.
Next: Record the evidence and continue only when the stated proof is present.
03 Establish least-privileged administration Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Create named admin accounts with SSH keys and scoped sudo; verify from a second session. Disable or restrict password/root login only after effective-config tests.
Read the result: Two independent authorized access paths work; unauthorized password/root tests fail.
Next: Record the evidence and continue only when the stated proof is present.
04 Patch and control software sources Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Use official or justified repositories, update packages, review removals, set unattended security policy, schedule reboots, and record exceptions.
Read the result: Security updates apply without broken dependencies and running kernel state is known.
Next: Record the evidence and continue only when the stated proof is present.
Procedure continued 05 Restrict network and services Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Remove/disable unused services, bind internal listeners appropriately, configure provider firewall plus nftables/UFW from required-flow matrix, and rate-limit exposed authentication.
Read the result: External probes show only documented ports and required internal flows still work.
Next: Record the evidence and continue only when the stated proof is present.
06 Add isolation, logging, and monitoring Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Use dedicated service users, restrictive files, systemd protections where compatible, NTP, persistent/remote logs as needed, health checks, resource alerts, and audit coverage.
Read the result: Service failure, auth attempts, resource pressure, and clock state are observable.
Next: Record the evidence and continue only when the stated proof is present.
07 Prove backup, recovery, and handoff Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Back up data/config/secrets references, restore into isolation, test application journeys and console recovery, then document update, restart, rollback, and incident steps.
Read the result: A clean operator restores the minimum viable service within declared targets.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet Evidence record Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Provider, region, image, Ubuntu release/kernel, IPs, DNS, and ownership.
- Console/rescue access, authorized administrators, SSH keys, and recovery contacts.
- Listening sockets, processes, packages, services, users, groups, sudo, and scheduled jobs.
- Firewall/security-group rules, routes, DNS, NTP, logging, and update policy.
- Backup scope, snapshot policy, restore proof, service health, and rollback checkpoint.
Acceptance scoreboard
- Provider ownership, console, billing, IP/DNS, and recovery contacts are documented.
- Release, packages, services, users, jobs, sockets, firewall, and time baseline is preserved.
- Named key-based admin and independent recovery access are proven before restrictions.
- Security updates, reboot policy, and repository trust are controlled.
- External probes expose only required ports; service users and files are least-privileged.
- Monitoring, backups, isolated restore, rollback, and operator runbook pass.
Decision rule SHIP / AUTOMATE GATE Proceed only when every required acceptance check is supported by direct evidence, rollback is available, and the remaining risk is explicitly owned. Unknown is not a pass.
Minimum handoff record
- Versioned ubuntu vps production baseline scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Starting Problem:
A developer enables UFW deny and disables password SSH in one session, then loses access because the cloud image never installed the intended key for the new user.
Evidence collected
- No provider console test was performed.
- The original root session was the only access path.
- Effective sshd config was not checked.
- No snapshot or rollback record exists.
Decision The sequence was unsafe. Use provider rescue/console, restore authorized access, then repeat changes one at a time with a second session.
Actions taken
- Recovered through provider console.
- Verified key ownership and sshd effective config.
- Tested new admin in parallel session.
- Applied firewall and auth changes separately with external probes.
Proof Of Completion:
Two named admin key sessions and console recovery work; password/root policy matches intent; only required ports are externally reachable.
Why this example matters The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Provider ownership, console, billing, IP/DNS, and recovery contacts are documented.
- Release, packages, services, users, jobs, sockets, firewall, and time baseline is preserved.
- Named key-based admin and independent recovery access are proven before restrictions.
- Security updates, reboot policy, and repository trust are controlled.
- External probes expose only required ports; service users and files are least-privileged.
- Monitoring, backups, isolated restore, rollback, and operator runbook pass.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear What happened What it usually means Next safe move
Public port open despite UFW Traffic may bypass expected layer or Compare provider firewall, nftables, sockets, and external probe deny service listens on another interface/port.
SSH key rejected after Include order, permissions, user, key, Use console to inspect effective sshd config and logs hardening or algorithm policy is wrong.
Updates installed but reboot Patched packages are not active until Compare running kernel and reboot-required state pending controlled restart.
Disk grows unexpectedly Retention or runaway service can make Inspect journal, application logs, cache, and inode use the host unavailable.
Reusable handoff record
- Versioned ubuntu vps production baseline scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
Field Type Requirement
target object Versioned environment, resource, identity, or workflow being evaluated.
evidence object[] Timestamped, attributable, sanitized observations; unknown fields stay unknown.
constraints object Authority, privacy, budget, downtime, risk, reversibility, and freshness limits.
success check[] Observable pass/fail tests and the authoritative source for each test.
Returned output
Field Type Requirement
diagnosis object Likely layer, supporting and conflicting evidence, alternatives, and confidence.
plan step[] Ordered bounded actions with owner, risk, expected proof, and stop condition.
verification check[] Observed pass/fail/unknown results, not inferred success from command exit alone.
handoff object Sanitized evidence record, recovery state, remaining risk, and next review trigger.
Agent refusal and escalation rules
•
Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
•
Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
•
Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
•
Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule
Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low
confidence; a controlled test that isolates the layer and passes verification can support high confidence.Official reference starting points
- https://documentation.ubuntu.com/server/
- https://ubuntu.com/security/notices
- https://www.cisecurity.org/benchmark/ubuntu_linux