Trace a request through process listeners, host firewall, cloud firewall, reverse proxy, TLS termination, and upstream application routing.
The result you're building
A verified request path from external client through provider and host firewalls, TLS proxy, and the intended local application listener, exposing only required ports and preserving correct client/scheme headers.
Use this guide when
- A local service works but the public hostname/port fails, times out, or returns 502/504.
- You need to expose a new service without accidentally publishing its origin/admin port.
Do not use it as a substitute for
- Do not open broad port ranges or bind admin services to all interfaces as a diagnostic shortcut.
- Do not trust forwarded client-IP headers from arbitrary internet clients.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Public hostname/IP/port and intended protocol.
- Application bind address/port and owning process.
- Host firewall plus cloud/provider security rules.
- Proxy virtual host/upstream/TLS config and external test location.
Understand the system before fixing it
A port is reachable only when every hop permits it
Process listener, bind address, routing/NAT, host firewall, provider firewall, proxy, and TLS must agree.
Bind address sets exposure
127.0.0.1 limits to host; 0.0.0.0 exposes on reachable interfaces. Put public exposure at the proxy when possible.
502 and timeout differ
502 proves a proxy answered but upstream response failed; timeout may mean packet drop, route, or listener absence.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Local origin refuses | Listener/app | ss -ltnp and local curl | Fix service/start/bind/port before firewall. |
| Local works; LAN/public timeout | Firewall/routing | Check host/provider rules and packet path | One network layer drops/does not route. |
| Public returns 502 | Proxy upstream | Compare upstream URL/protocol with listener and proxy log | Wrong port/protocol/name, permissions, or app reset. |
| Wrong site/cert | Virtual host/SNI | curl with Host/SNI and inspect active config | Request reaches default vhost/certificate. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Prove the local listener
Why: Firewall edits cannot create a listening process.
Do: Identify PID, address, port, protocol, and local health.
sudo ss -ltnp
curl -v http://127.0.0.1:PORT/healthRead the result: No listener means app config/service. Loopback-only may be correct when proxy is local.
Next: Record expected exposure.
Step 02 — Test each network boundary
Why: One end-to-end test cannot identify the dropping hop.
Do: Test from host, same LAN/VPC, and external client; inspect provider and UFW/nftables rules.
sudo ufw status numbered
sudo nft list rulesetRead the result: First transition from success to failure locates the boundary.
Next: Open only required source/destination/protocol.
Step 03 — Validate proxy configuration
Why: A syntactically invalid or wrong vhost can leave old config active.
Do: Inspect full active config, Host match, TLS binding, upstream protocol/address, and forwarding headers; run config test.
sudo nginx -T 2>/dev/null
sudo nginx -t
sudo systemctl reload nginxRead the result: Duplicate/default vhosts and http/https upstream mismatch are common.
Next: Reload, do not restart, after successful validation.
Step 04 — Protect origin and forwarded identity
Why: Public origin bypass and spoofed headers defeat proxy controls.
Do: Bind origin to loopback/private interface or firewall it to proxy; trust forwarded headers only from known proxy addresses.
Read the result: Direct external connection to origin should fail when not intended.
Next: Verify app sees correct scheme/client through proxy.
Step 05 — Test TLS, redirects, and failures
Why: Happy-path 200 misses loops and unsafe fallbacks.
Do: Test HTTP, HTTPS, unknown host, large/slow request within policy, upstream stopped, and external origin reachability.
Read the result: Expected status and no information leak define safe behavior.
Next: Restore upstream and retest.
Step 06 — Document exposure
Why: Future changes can reopen retired services.
Do: Record listeners, firewall rules, provider rules, vhosts, certificates, ownership, and external scans for intended ports.
Read the result: Only documented ports should be reachable.
Next: Schedule rule review.
Worked example
Evidence collected
- nginx answers with valid certificate.
- Proxy error log shows connection refused to 127.0.0.1:8080.
- Application listens on 3000.
- Firewall is irrelevant because proxy and origin share host.
Decision: The proxy upstream port is stale.
Actions taken
- Backed up and corrected only upstream port.
- Ran nginx config test and reload.
- Retested local origin and public HTTPS.
Why this example matters: Hop-by-hop testing avoided opening ports that could never fix a local proxy mismatch.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Intended process owns expected listener.
- Only intended public ports are reachable externally.
- Proxy routes correct Host/SNI to correct upstream.
- Origin is not unintentionally exposed.
- Forwarded identity/scheme are trusted only from proxy.
- Failure responses and logs are bounded and useful.
Rollback or safe recovery
- Restore proxy backup and reload after validation.
- Remove the exact newly added firewall/provider rule if exposure is wrong.
- Rebind origin to prior local address and verify admin access.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| UFW allows but port closed | No listener/provider/NAT still blocks. | Return to listener and upstream boundary. |
| 502 after reload | Upstream mismatch or app failure. | Read proxy error plus local curl. |
| Client IP always proxy | Forwarded-header trust/config incomplete. | Configure known proxy range and app proxy trust. |
| External scan shows extra port | Origin/daemon bound publicly or old rule remains. | Identify PID/rule, then close narrow exposure. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Listener/process/bind and local health.
- Host/provider firewall rules and test points.
- Active proxy/vhost/upstream/TLS config.
- External reachability and failure-mode tests.
- Change, rollback, and exposure inventory.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points