Agent Memory and State Integrity

Design agent memory that is attributable, scoped, correctable, erasable, and resistant to instruction poisoning or cross-user leakage.

The result you're building

A memory system that separates conversation state, user preferences, task facts, credentials, and durable knowledge; records provenance and consent; resolves conflicts; and proves isolation, correction, deletion, and recovery.

Use this guide when

• An assistant stores facts or preferences across sessions.
• Several agents or workflows share task state.
• Bad or stale memory could cause spending, messaging, access, or safety failures.

Do not use it as a substitute for

• Saving every message indefinitely as 'memory.'
• Treating retrieved text as trusted instruction or using memory as a secret store.

Before you change anything

• Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
• Memory classes, owners, scopes, retention, and permitted consumers.
• Source message/event, observation time, confidence, and last validation.
• Conflict, correction, deletion, and legal-hold behavior.
• Encryption, tenant isolation, access logs, and export path.
• Poisoning, cross-user, stale-state, deletion, restore, and replay test results.

Understand the system before fixing it

Capability is not authority
A tool may be technically able to act while policy, budget, ownership, or user intent still forbids the action. Keep execution permission explicit and revocable.

Deterministic boundaries contain probabilistic reasoning
Models may propose plans, but schemas, allowlists, budgets, approvals, idempotency, and postconditions must decide what can actually happen.

Memory is evidence, not authority
A remembered preference can inform a proposal but cannot replace current authentication or approval for a sensitive action.

Deletion must reach derived indexes
Removing a source row while embeddings, summaries, caches, or backups remain active is not effective deletion.

Evidence-to-decision map

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

Step 01 — Classify memory and its purpose

Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: Define ephemeral context, task state, durable user preference, verified fact, organizational knowledge, and prohibited secret classes. Record why each class exists.

Read the result: Every stored item has one class, owner, scope, retention rule, and allowed use.

Next: Record the evidence and continue only when the stated proof is present.

Step 02 — Attach provenance and confidence

Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.

Do: Store source event, actor, time, extraction method, confidence, validation time, and links to corrections or superseding records.

Read the result: A reviewer can trace every durable claim to its source.

Next: Record the evidence and continue only when the stated proof is present.

Step 03 — Enforce tenant and task isolation

Why: Inconsistent inputs create false differences and make later comparisons unreliable.

Do: Use server-side identity scopes in every primary, vector, cache, and summary query. Test missing and manipulated tenant fields.

Read the result: Cross-tenant canaries never appear in results, logs, or exports.

Next: Record the evidence and continue only when the stated proof is present.

Step 04 — Treat retrieved memory as untrusted data

Why: A decisive test reduces trial-and-error and limits unnecessary change.

Do: Separate memory from system/developer instruction, escape tool-like content, and require policy checks before acting on recalled facts.

Read the result: A poisoned note cannot alter tool policy, reveal secrets, or authorize action.

Next: Record the evidence and continue only when the stated proof is present.

Step 05 — Resolve conflicts and staleness

Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.

Do: Prefer current authenticated instruction, keep superseded history, set expiry/revalidation triggers, and return uncertainty when claims disagree.

Read the result: The system explains which record won and why.

Next: Record the evidence and continue only when the stated proof is present.

Step 06 — Implement correction, export, and deletion

Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.

Do: Propagate changes through summaries, embeddings, caches, replicas, and retention-aware backups; record completion and exceptions.

Read the result: A test identity can be corrected, exported, and removed from active retrieval.

Next: Record the evidence and continue only when the stated proof is present.

Step 07 — Run integrity and restore tests

Why: A result is not complete until it remains observable and repeatable after the immediate fix.

Do: Inject cross-user, stale, contradictory, poisoned, and deleted canaries; restore a backup into isolation and confirm policy survives.

Read the result: Scheduled tests prove isolation and lifecycle behavior end to end.

Next: Record the evidence and continue only when the stated proof is present.

Operational worksheet

Evidence record

• Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
• Memory classes, owners, scopes, retention, and permitted consumers.
• Source message/event, observation time, confidence, and last validation.
• Conflict, correction, deletion, and legal-hold behavior.
• Encryption, tenant isolation, access logs, and export path.
• Poisoning, cross-user, stale-state, deletion, restore, and replay test results.

Acceptance scoreboard

• Every durable item has owner, scope, purpose, source, time, confidence, retention, and permitted use.
• Current authenticated instruction and authoritative state outrank stale summaries.
• Cross-tenant and cross-task isolation tests pass for primary, vector, and cache paths.
• Retrieved memory cannot change policy or authorize sensitive action.
• Correction, export, and deletion reach derived indexes and caches.
• Backup restore preserves current isolation and lifecycle rules.

Minimum handoff record

• Versioned agent memory and state integrity scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Worked example

Evidence collected

• The CRM row shows opt-out.
• The memory summary still says follow-up permitted.
• Retrieval ranks the summary above the CRM event.
• Deletion jobs update SQL but not the vector index.

Decision: The system has stale derived state and a dangerous precedence rule. Current authoritative consent must block outreach regardless of older memory.

Actions taken

• Marked consent as authoritative structured state.
• Removed the stale summary and rebuilt affected vectors.
• Added opt-out enforcement at the send boundary.
• Added correction and deletion propagation tests.

Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.

Verify, recover, and hand off

Completion tests

• A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
• Every durable item has owner, scope, purpose, source, time, confidence, retention, and permitted use.
• Current authenticated instruction and authoritative state outrank stale summaries.
• Cross-tenant and cross-task isolation tests pass for primary, vector, and cache paths.
• Retrieved memory cannot change policy or authorize sensitive action.
• Correction, export, and deletion reach derived indexes and caches.
• Backup restore preserves current isolation and lifecycle rules.

Rollback or safe recovery

• Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
• Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
• Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
• Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.

If the expected result does not appear

Reusable handoff record

• Versioned agent memory and state integrity scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Agent delivery contract

Required inputs

Agent refusal and escalation rules

• Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
• Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
• Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
• Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.

Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.

Official reference starting points

• https://www.nist.gov/privacy-framework
• https://genai.owasp.org/llmrisk/llm01-prompt-injection/
• https://www.w3.org/TR/prov-overview/