SAYLOR Use caches without serving cross-user, stale, unauthorized, corrupted, or stampeding responses by making keys, freshness, invalidation, and fallback explicit.
The result you're building
A cache contract that defines what is cacheable, binds entries to identity/version/authorization, exposes age, prevents stampedes, verifies invalidation, and fails safely when cache or origin is unhealthy.
Use this guide when
- You cache API results, sessions, pages, model outputs, market data, or database objects.
- Staleness or wrong-tenant data could change decisions.
- Origin load spikes when keys expire or cache fails.
Do not use it as a substitute for
- Caching authenticated responses under a URL-only key.
- Using flush-all as the normal invalidation or recovery method.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Resource semantics, authority, sensitivity, volatility, and permitted staleness.
- Complete key dimensions: tenant/user/role/query/version/locale/authorization/data version.
- TTL, stale-while-revalidate, negative cache, tag/event invalidation, and delete behavior.
- Serializer/schema/compression/version and integrity checks.
- Hit/miss/age/eviction/stampede/leak/invalidation/failover test evidence.
Understand the system before fixing it
Observe before mutating
Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.
Recovery must be exercised
A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.
A cache key is a security boundary
If two requests can produce different authorized results, every relevant identity, role, permission, locale, version, and input dimension must be represented or caching must be disabled.
TTL is not complete invalidation
Corrections, revocations, deletes, and security changes may require immediate tag/event invalidation plus sweeps for missed events.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| User sees another tenant's response | Key/authorization | Compare key dimensions and request principals | Tenant or permission context is missing from the key. |
| Old schema crashes reader | Serialization/version | Inspect entry version and deployment overlap | Writers/readers changed without namespaced compatibility. |
| Origin collapses at expiry | Stampede | Plot concurrent misses for hotkey | No single-flight, jitter, prewarm, or stale policy exists. |
| Deleted item remains served | Invalidation | Trace delete event/tag and replica/cache layers | Deletion did not reach all active cache paths. |
| Cache outage makes service wrong | Fallback | Force cache timeout and observe origin/fail mode | The application treats cache error as data or bypasses safety limits. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Classify cacheable results
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: For each route/value, define sensitivity, identity dependence, volatility, source cost, correctness requirement, and maximum age. Disable caching when safe keying is impractical.
Read the result: Every cache has an owner and explicit staleness contract.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Design complete versioned keys
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Include normalized inputs, tenant/user/role or authorization digest, resource/data/schema version, locale, and output variant. Hash large sensitive components safely.
Read the result: Requests that can differ never collide; equivalent requests share deliberately.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Define freshness and invalidation
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Set TTL from fact volatility, add jitter, declare stale-while-revalidate only where safe, and use tag/event invalidation for writes, revocations, corrections, and deletion.
Read the result: A changed authoritative record becomes visible within the declared window.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Control serialization and integrity
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Namespace formats, validate type/schema/version on read, cap size/decompression, encrypt sensitive values if caching is allowed, and reject corrupt entries.
Read the result: Old, malformed, oversized, or wrong-type entries fail as misses, not trusted data.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Prevent stampedes and cache abuse
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Use single-flight/leases, request coalescing, prewarm hot keys, bounded negative caching, quotas, and protection against attacker-controlled key explosion.
Read the result: Hot-key expiry and random-key load stay within origin capacity.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Design fallback and failure mode
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Set cache timeouts, circuit state, origin budgets, and explicit stale/error behavior. Never treat cache connection error as 'not found' for authorization.
Read the result: Cache loss degrades according to the contract without leaking or lying.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Test isolation and invalidation
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Run cross-tenant canaries, permission changes, delete/correct, schema overlap, hot-key expiry, cache restart, replica lag, and missed-event sweep.
Read the result: No leak occurs and freshness, fallback, and recovery match published behavior.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Resource semantics, authority, sensitivity, volatility, and permitted staleness.
- Complete key dimensions: tenant/user/role/query/version/locale/authorization/data version.
- TTL, stale-while-revalidate, negative cache, tag/event invalidation, and delete behavior.
- Serializer/schema/compression/version and integrity checks.
- Hit/miss/age/eviction/stampede/leak/invalidation/failover test evidence.
Acceptance scoreboard
- Cacheability decision covers sensitivity, authority, volatility, cost, and permitted age.
- Keys include every result-changing input and authorization dimension.
- Writes, revocations, corrections, and deletes propagate through active caches.
- Serializer/schema/version and integrity are validated before use.
- Stampede, random-key, negative-cache, and origin capacity controls pass.
- Cross-tenant, schema overlap, outage, restart, invalidation, and sweep tests pass.
Minimum handoff record
- Versioned cache integrity and safe invalidation scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
/profile by path and serves one customer's profile to another.Evidence collected
- The route is authenticated.
- Cache key contains only method and URL.
- Reverse proxy caches 200 responses by default.
- Response lacks a private/no-store policy.
Decision: This is a cross-user data incident. Disable affected caching, preserve logs, notify the owner, and redesign keys/policies before reactivation.
Actions taken
- Purged and disabled the unsafe cache rule.
- Added private/no-store where shared caching is not justified.
- Included tenant/user/authorization version in approved caches.
- Added cross-user canary tests.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Cacheability decision covers sensitivity, authority, volatility, cost, and permitted age.
- Keys include every result-changing input and authorization dimension.
- Writes, revocations, corrections, and deletes propagate through active caches.
- Serializer/schema/version and integrity are validated before use.
- Stampede, random-key, negative-cache, and origin capacity controls pass.
- Cross-tenant, schema overlap, outage, restart, invalidation, and sweep tests pass.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| User sees another tenant's response | Tenant or permission context is missing from the key. | Compare key dimensions and request principals |
| Old schema crashes reader | Writers/readers changed without namespaced compatibility. | Inspect entry version and deployment overlap |
| Origin collapses at expiry | No single-flight, jitter, prewarm, or stale policy exists. | Plot concurrent misses for hot key |
| Deleted item remains served | Deletion did not reach all active cache paths. | Trace delete event/tag and replica/cache layers |
Reusable handoff record
- Versioned cache integrity and safe invalidation scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points