SAYLOR Recover SSH access without weakening the host broadly, then harden authentication through tested effective configuration and multiple recovery paths.
The result you're building
A recovered SSH service with known-good provider console access, verified identity/key/permissions/network path, validated effective sshd configuration, controlled authentication policy, and a lockout-safe change procedure.
Use this guide when
- SSH reports timeout, refusal, disconnect, or permission denied.
- A recent key, user, firewall, port, or sshd change caused lockout.
- You need to harden SSH while retaining recoverability.
Do not use it as a substitute for
- Opening SSH to the world with passwords or copying private keys between systems.
- Restarting/reinstalling sshd repeatedly before identifying whether the failure is network, service, or authentication.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Exact client command/error, source network, target IP/host/port, and time.
- Provider console/rescue availability and cloud firewall rules.
- Client key fingerprint, agent identities, known_hosts result, and verbose SSH trace.
- Server sockets, firewall, journal/auth logs, user/shell/home/key permissions.
sshd -tandsshd -T -Ceffective configuration for the target user/address.
Understand the system before fixing it
Observe before mutating
Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.
Recovery must be exercised
A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.
The error class narrows the layer
Timeout points to route/firewall; refusal to listener/service; disconnect to policy/load; permission denied to identity/authentication. Preserve the exact message.
Included configuration can override assumptions
Read the effective configuration for the actual user/address/host, not only one file in /etc/ssh.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Connection times out | Network/firewall | Probe target port and compare cloud/host rules | Packets are filtered, routed wrong, or target/port is incorrect. |
| Connection refused | Listener/service | Check ss -lntp and systemctl status ssh | No service listens on that address/port or it failed to start. |
| Permission denied (publickey) | Identity/auth | Run ssh -vvv and inspect server auth log | Wrong user/key, agent selection, permissions, or policy rejection. |
| Host key changed warning | Identity/integrity | Verify new fingerprint through trusted console/provider | Server was rebuilt/changed or connection may be intercepted; do not delete blindly. |
| Works for one source only | Match/firewall | Evaluate sshd -T -C and source rules | Conditional config, allowlist, fail2ban, or network policy differs. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Classify the failure from the client
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Record DNS/IP, port, user, exact message, and ssh -vvv identity/negotiation output. Check whether another admin/session/source works.
Read the result: The investigation begins at the most likely network, service, host-key, or auth layer.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Secure a recovery channel
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Verify provider console, serial, rescue, or authorized second session. Snapshot config and record current sockets/firewall before changes.
Read the result: Recovery remains possible if the next SSH change fails.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Verify network and listener
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Confirm provider firewall, route, host firewall, correct address/port, ss listener, service status, config syntax, and journal startup errors.
Read the result: A listener is reachable on exactly the intended path.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Verify user and key chain
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Check account lock/expiry/shell/home, .ssh and authorized_keys ownership/modes, key line/options, client fingerprint, ssh-agent choices, and server auth logs.
Read the result: The server sees the expected key for the expected user and states why it accepts/rejects it.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Inspect effective policy
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Run syntax validation and effective configuration for target connection; review Include, Match, AllowUsers/Groups, AuthenticationMethods, algorithms, and root/password settings.
Read the result: Effective values match documented intent before reload.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Apply one reversible change
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Back up config, edit the narrow cause, validate syntax, reload rather than blindly restart, and test from a new session while keeping the old one open.
Read the result: New session succeeds and existing session remains available.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Harden and regression-test
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Use named users, keys, least privilege, provider/host allowlists where practical, rate limits, logging, key rotation, and tested console recovery. Test denied cases too.
Read the result: Allowed identities work; wrong user/key/source and prohibited methods fail as designed.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Exact client command/error, source network, target IP/host/port, and time.
- Provider console/rescue availability and cloud firewall rules.
- Client key fingerprint, agent identities, known_hosts result, and verbose SSH trace.
- Server sockets, firewall, journal/auth logs, user/shell/home/key permissions.
sshd -tandsshd -T -Ceffective configuration for the target user/address.
Acceptance scoreboard
- Exact error and verbose trace identify the likely failure layer.
- Provider console or a second privileged session remains available during change.
- DNS, route, cloud firewall, host firewall, listener, and service status agree.
- User, key fingerprint, permissions, agent choice, and server auth log reconcile.
- Syntax and connection-specific effective configuration are validated.
- Allowed and denied regression tests plus host-key/recovery documentation pass.
Minimum handoff record
- Versioned ssh lockout recovery and hardening scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
sshd -tpasses.- Server listens on 2222 locally.
- UFW allows 2222.
- Provider firewall still allows only TCP 22.
Decision: The host service is healthy; the provider network layer blocks the new port. Update the provider rule from console while preserving port 22 until the new path is proven.
Actions taken
- Added scoped provider rule for 2222.
- Probed externally from expected source.
- Opened a second session on 2222.
- Removed old port only after recovery documentation and tests.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Exact error and verbose trace identify the likely failure layer.
- Provider console or a second privileged session remains available during change.
- DNS, route, cloud firewall, host firewall, listener, and service status agree.
- User, key fingerprint, permissions, agent choice, and server auth log reconcile.
- Syntax and connection-specific effective configuration are validated.
- Allowed and denied regression tests plus host-key/recovery documentation pass.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Connection times out | Packets are filtered, routed wrong, or target/port is incorrect. | Probe target port and compare cloud/host rules |
| Connection refused | No service listens on that address/port or it failed to start. | Check ss -lntp and systemctl status ssh |
| Permission denied (publickey) | Wrong user/key, agent selection, permissions, or policy rejection. | Run ssh -vvv and inspect server auth log |
| Host key changed warning | Server was rebuilt/changed or connection may be intercepted; do not delete blindly. | Verify new fingerprint through trusted console/provider |
Reusable handoff record
- Versioned ssh lockout recovery and hardening scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points