SAYLOR Protect clients and services from overload with explicit quotas, concurrency, queues, retry guidance, fairness, and load-shedding behavior.
The result you're building
A tested capacity and backpressure policy that admits bounded work, returns useful retry signals, prevents retry storms, protects priority operations, and degrades without losing or duplicating accepted jobs.
Use this guide when
- An API calls costly models, databases, blockchains, or third-party providers.
- Traffic arrives in bursts or through many agent workers.
- Overload currently appears as timeouts, 5xx errors, or provider bans.
Do not use it as a substitute for
- Increasing timeouts or automatic retries without measuring capacity.
- Using one global limit that lets a single tenant or expensive route consume the system.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Per-route work units, cost, concurrency, latency, and dependency limits.
- Client/tenant identity, quotas, priority classes, and fairness policy.
- Queue depth/age, worker capacity, timeout, and cancellation behavior.
- 429/503 response schema, Retry-After, idempotency, and client backoff.
- Steady, burst, retry-storm, dependency-slowdown, and recovery load tests.
Understand the system before fixing it
Provenance is part of the record
A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.
Schema changes are product changes
Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.
Rate and concurrency are different
Requests per second does not bound simultaneous expensive work. Protect CPU, memory, database connections, provider quotas, and queue age directly.
Retries are new load
Clients and workers must use bounded exponential backoff with jitter and honor server guidance; otherwise a small failure becomes a synchronized storm.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Latency spikes before CPU maxes | Dependency/concurrency | Trace active work and connection/provider saturation | A downstream pool or serialized resource is the bottleneck. |
| 429 clients retry immediately | Contract/client | Inspect Retry-After and retry jitter | Server guidance is missing or clients ignore it. |
| One tenant starves others | Fairness | Break usage by tenant/route/work unit | Global first-come queue lacks isolation or weights. |
| Queue grows after recovery | Admission | Compare arrival, service rate, and queue age | System accepts more work than it can drain. |
| Timeout repeats side effect | Idempotency | Replay request key after ambiguous timeout | Retry safety is missing at the operation boundary. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Measure work and bottlenecks
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Define request cost units, critical dependencies, p50/p95 service time, connection use, memory, and external quotas by route and tenant.
Read the result: Capacity model identifies the first constrained resource.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Set admission and concurrency policy
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Apply tenant/route token buckets or quotas plus hard in-flight limits. Reserve capacity for health, cancellation, and priority operations.
Read the result: Burst tests stay within protected resource thresholds.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Bound queues by age and size
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Choose synchronous rejection or durable queue, set maximum depth/age/deadline, and reject work that cannot complete in time.
Read the result: Accepted jobs have a plausible completion window; rejected jobs consume little work.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Publish retry-safe responses
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Return stable 429/503 errors, Retry-After where appropriate, correlation ID, quota state as safe, and idempotency guidance. Do not leak other tenants.
Read the result: A compliant client can retry without guessing or duplicating work.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Implement client backoff and cancellation
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Use exponential backoff with jitter, cap attempts/time, honor deadline and Retry-After, cancel obsolete work, and avoid hedging non-idempotent actions.
Read the result: Failure injection does not create a retry storm.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Shed optional work first
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Disable enrichment, reduce batch, serve explicit stale cache when allowed, or return partial with omissions. Preserve auth, safety, and correctness gates.
Read the result: Degraded responses remain schema-valid and policy-compliant.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Load-test recovery and fairness
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Test steady, burst, slow dependency, brownout, retry storm, tenant bully, worker loss, and recovery. Alert on queue age and rejected/accepted outcomes.
Read the result: System recovers without a second overload wave or silent accepted-work loss.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Per-route work units, cost, concurrency, latency, and dependency limits.
- Client/tenant identity, quotas, priority classes, and fairness policy.
- Queue depth/age, worker capacity, timeout, and cancellation behavior.
- 429/503 response schema, Retry-After, idempotency, and client backoff.
- Steady, burst, retry-storm, dependency-slowdown, and recovery load tests.
Acceptance scoreboard
- Capacity model covers work units, concurrency, dependencies, queues, and tenant fairness.
- Admission rejects work before protected resources saturate.
- Queue size and age are bounded by completion deadline.
- 429/503 responses and clients implement consistent bounded retry guidance.
- Idempotency prevents retry duplication for side-effecting work.
- Brownout, retry-storm, bully-tenant, worker-loss, and recovery tests pass.
Minimum handoff record
- Versioned api rate limit and backpressure control scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- Provider p95 rises from 300 ms to 8 s.
- Server allows unbounded in-flight calls.
- Clients retry on timeout with no jitter.
- Queue age is not measured.
Decision: The outage is amplified by missing concurrency and retry control. Reject early, bound in-flight provider calls, and coordinate retry timing.
Actions taken
- Added per-provider concurrency and circuit state.
- Returned 503 with Retry-After and stable error code.
- Added jittered capped client retry.
- Shed optional enrichment and measured queue age.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Capacity model covers work units, concurrency, dependencies, queues, and tenant fairness.
- Admission rejects work before protected resources saturate.
- Queue size and age are bounded by completion deadline.
- 429/503 responses and clients implement consistent bounded retry guidance.
- Idempotency prevents retry duplication for side-effecting work.
- Brownout, retry-storm, bully-tenant, worker-loss, and recovery tests pass.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Latency spikes before CPU maxes | A downstream pool or serialized resource is the bottleneck. | Trace active work and connection/provider saturation |
| 429 clients retry immediately | Server guidance is missing or clients ignore it. | Inspect Retry-After and retry jitter |
| One tenant starves others | Global first-come queue lacks isolation or weights. | Break usage by tenant/route/work unit |
| Queue grows after recovery | System accepts more work than it can drain. | Compare arrival, service rate, and queue age |
Reusable handoff record
- Versioned api rate limit and backpressure control scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points