Turn a failed API request into a corrected, reproducible request by separating transport, authentication, validation, authorization, rate-limit, and server failures.
The result you're building
A minimal reproducible request, a layer-specific diagnosis, a corrected request or server behavior, and an automated regression test that distinguishes transport, authentication, authorization, validation, throttling, and server faults.
Use when: an HTTP/API call fails, differs between curl and browser, or works in one environment only; a third-party API returns ambiguous 4xx/5xx; you need to hand an exact reproducible failure to an API owner.
Not a substitute for: publishing credentials, full tokens, cookies, or customer payloads in a support transcript; retrying side-effecting requests until one appears successful.
Before you change anything, collect: method, fully resolved URL, sanitized headers, content type, exact body bytes; timestamp, response status, headers/body, correlation/request ID; API doc version, environment, account/tenant, expected scope; whether the operation is read-only, idempotent, or side-effecting; a way to reproduce with a low-risk test resource.
Understand the system first
- Status class narrows the owner, not the exact cause — 3xx redirects, 4xx request/auth errors, 429 throttling, 5xx server/gateway failures point to different owners. Read provider-specific codes and headers before changing the request.
- Browser and curl execute different security models — CORS is browser-enforced, not curl. Cookies, preflight OPTIONS, redirects, proxies, TLS stores, and compression also differ.
- The reproducible unit is raw HTTP — SDK errors can hide the method, URL, headers, body, retries, and response. Capture a sanitized wire-equivalent request before blaming the SDK.
Evidence-to-decision map
| Evidence | Likely layer | First check | Meaning |
|---|---|---|---|
| DNS/TLS/connect timeout | Transport | curl -v --connect-timeout 10 URL | No status at all — fix name resolution, cert, proxy, or reachability first |
| 401 | Authentication | WWW-Authenticate, token issuer/audience/expiry | Credential missing, invalid, expired, or wrong environment |
| 403 | Authorization/policy | Compare identity, resource, tenant, scopes, policy | Authenticated but denied for this action/resource |
| 400/404/405/415/422 | Request contract | Compare path, method, content type, schema | Doesn't match route/validation rules — don't rotate credentials first |
| 429 | Rate/cost | Read retry/reset/limit headers | Throttle per server guidance, don't tight-loop |
| 500/502/503/504 | Server/gateway | Capture request ID, test dependency/health | Provider/upstream failure; safe retry depends on idempotency |
Step-by-step procedure
01 Reduce to one sanitized request.
curl -sS -D headers.txt -o body.txt -w '%{http_code} %{time_total}\n' \
-X METHOD 'https://api.example.com/path' \
-H 'Authorization: Bearer REDACTED' -H 'Content-Type: application/json' \
--data-binary @request.json02 Classify the failing layer before editing. Use whether a status exists, its class, error code, headers, and request ID to pick exactly one branch. Generic 500 bodies aren't enough — provider logs need the correlation ID.
03 Validate the request contract byte-for-byte.
jq -e . request.json
curl -i -X OPTIONS 'https://api.example.com/path'
sha256sum request.json415 → media type; 405 → method/route; 422 → valid syntax but unacceptable semantics.
04 Validate identity and authorization separately. Decode only non-secret token claims locally, compare issuer/audience/expiry/scopes, verify the resource belongs to the same account/environment. 401 = credential validation failed; 403 = authenticated identity lacks permission.
05 Control retries, capture the provider signal. For 429/5xx use bounded exponential backoff with jitter and the provider's retry header; reuse idempotency keys for the same logical operation.
06 Turn the fix into a regression test. A fixture with sanitized inputs and assertions for status, schema, error mapping, timeout, and idempotent retry — it must fail on the original bug and pass on the fix.
Worked example
A browser POST returns a CORS error while the same token/JSON work in curl. curl gets 201; the browser sends an OPTIONS preflight; the preflight response lacks Access-Control-Allow-Origin and allowed headers; the POST never reaches the app log. Decision: auth and schema are fine — the browser is blocked at the CORS preflight boundary. Fix: added the exact trusted frontend origin, methods, and required headers at the gateway; kept credentials disabled for wildcard origins; tested OPTIONS/POST from allowed and disallowed origins. Proof: allowed origin completes preflight and POST; disallowed stays blocked; curl unchanged.
Verify, recover, hand off
Complete when: the minimal sanitized request returns the documented success status/schema; invalid auth, insufficient permission, and invalid payload each return distinct stable errors; retries obey idempotency/rate-limit rules; browser preflight/credential policy allows only intended origins; a regression test reproduces the old failure and proves the fix.
Rollback: restore prior gateway/API config if validation or unrelated routes regress; disable the new path/feature flag rather than resending uncertain side effects; revoke test credentials created during diagnosis.
| Symptom | Usually means | Next move |
|---|---|---|
| curl works, SDK fails | SDK changes URL/serialization/auth/proxy/TLS/retry | Enable sanitized wire logging, compare request bytes |
| 401 becomes 403 | Identity valid but lacks resource permission | Inspect scope/tenant/ownership/policy, don't keep rotating tokens |
| Occasional 500 with no request ID | Insufficient observability | Add client correlation ID/timestamp, capture headers + bounded retry outcome |
| Duplicate object after timeout | Retry lacked stable idempotency/reconciliation | Stop retries, find by idempotency/reference, fix client semantics |
Handoff record: sanitized raw request/response with timestamp and request ID; failure classification and evidence ruling out adjacent layers; corrected request/config diff; retry/idempotency decision and side-effect reconciliation; automated regression test and remaining provider dependencies.
For agents
Inputs: request (method/URL/sanitized headers/raw body hash/environment), response (status/sanitized headers/body/latency/timestamp/request ID), contract (documented route/auth/schema/retry semantics), sideEffect (none/idempotent/non-idempotent/unknown). Output: diagnosis, plan, verification, handoff. Same refusal/escalation/confidence rules as the series.
References
- https://developer.mozilla.org/en-US/docs/Web/HTTP
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS
- https://www.rfc-editor.org/rfc/rfc9110