AI agent memory needs its own security boundary

Why memory changes the threat model

A stateless prompt injection has to affect the current interaction. Persistent memory gives the attack another path: enter through one source, survive the session, reappear in a different context, and influence a later decision. The person who sees the resulting behavior may never see the content that planted it.

A June 2026 arXiv preprint, From Untrusted Input to Trusted Memory, describes four memory-write channels and six attack classes in its study. Its benchmark found that agents designed to write and retrieve memory more aggressively were more exploitable in that test environment. This is a new preprint and not a universal population estimate, but it gives defenders a useful map of where to test.

A separate July 2026 preprint evaluated a two-stage attack called GhostWriter: first poison the memory, then wait for retrieval to activate it. The reported injection and activation rates belong to the authors' specific agents and setup, not every deployed system. The important operational lesson is the split itself. A write can appear harmless because the damaging behavior occurs later.

Microsoft reaches a similar architectural conclusion in its memory-security guidance: memory is both high-value user data and a control plane that shapes future behavior. That combination means storage security alone is insufficient. Integrity, provenance, authorization, and safe use at retrieval time all matter.

Map the whole memory dataflow

Start by drawing the actual path from an observed input to a later action. Teams often document the vector database while leaving the write policy, summarizer, retrieval filter, and tool authorization logic implicit. Those hidden transformations are where trust can change without a clear decision.

For every memory type, identify who or what can propose a write, which identity owns the record, how the system records its source, when it expires, which sessions can retrieve it, and whether retrieval can affect a tool call. Do this separately for explicit user preferences, inferred facts, task history, learned procedures, shared team knowledge, and developer files such as agent instructions.

• Input: email, document, webpage, repository, tool output, user statement, or model inference.
• Write decision: explicit user request, automatic extraction, summarization, or agent self-reflection.
• Storage: tenant, user, agent, namespace, integrity metadata, sensitivity, and retention period.
• Retrieval: query, ranking, scope filter, trust label, freshness check, and context placement.
• Action: answer generation, planning, tool selection, parameter construction, or approval request.
• Recovery: inspect, correct, quarantine, delete, restore, and invalidate dependent state.

Test the write boundary

The first gate decides whether untrusted content becomes durable state. A model should not be able to convert a sentence inside a document into a standing user preference merely because the sentence asks to be remembered. The write policy needs to evaluate current user intent, source provenance, sensitivity, tenant scope, and the type of memory being proposed.

Microsoft recommends gating writes on authorization, intent, and provenance. OWASP guidance similarly calls for malicious content to be sanitized, scoped, expired, or rejected before persistence. Neither recommendation means a classifier is a complete defense. A useful test suite assumes detection can fail and verifies that the resulting record still cannot cross identity or authority boundaries.

• Place an instruction to remember a false preference inside an email, document, webpage, and tool result. Verify that each source is treated as untrusted.
• Ask the agent to infer a durable fact from ambiguous evidence. Verify that uncertain inference is not silently promoted to trusted memory.
• Attempt to store credentials, access tokens, payment data, or regulated identifiers. Verify blocking or approved redaction before persistence.
• Repeat the same write across two users and two tenants. Verify that ownership and namespaces prevent cross-user or cross-tenant recall.
• Split a harmful instruction across several benign-looking writes. Verify that accumulation does not assemble a privileged procedure.
• Confirm that every accepted record carries source, actor, timestamp, policy decision, sensitivity, and expiry metadata.

Test retrieval and activation

A clean write gate reduces risk but cannot prove every stored record is safe. Memory may be wrong, stale, compromised through another path, or valid for one purpose and dangerous for another. The retrieval gate should therefore treat recalled content as evidence with provenance, not as a new system instruction.

Build delayed tests. Poison a low-risk session, end it, then start an unrelated task days or simulated sessions later. Change the wording of the retrieval query. Switch the available tools. Move from a read-only task to one with side effects. A memory defense that works only when the attack remains visible in the current transcript is not testing persistence.

• Retrieve poisoned content through semantic similarity, exact lookup, summarization, and agent-generated planning.
• Verify that expired, revoked, low-confidence, or wrong-scope memories are excluded before context construction.
• Test whether one retrieved record can override the user's current request, developer policy, or tool boundary.
• Test dormant triggers that activate only after a date, topic, tool, recipient, or workflow state changes.
• Confirm that retrieval-time filtering sees provenance and the original source, not only a compressed summary.
• Measure false positives and utility loss so a stricter policy does not quietly make memory unusable.

Keep memory separate from authority

The most important design rule is simple: memory can inform a decision, but it should not grant permission. A recalled note that says a user approved external sharing last month is not proof that the current file, recipient, and action are authorized now. Current identity, policy, scope, and user intent must be checked at the action boundary.

This separation limits the blast radius when a poisoned record survives both write and retrieval controls. The agent may produce a bad suggestion, but a separate authorization layer can still deny the tool call, constrain its parameters, or require fresh approval for an irreversible action.

• Never treat a stored instruction as a credential, consent receipt, or standing approval unless a dedicated policy system validates it.
• Bind approvals to the current user, action, resource, parameters, and expiration time.
• Give memory services no broader tool or data access than they need to store and retrieve scoped records.
• Require a new authorization decision when recalled content would trigger deletion, code execution, external communication, privilege change, or sensitive data movement.

Make failures observable and reversible

Memory poisoning is difficult to investigate when logs show only the final answer. Defenders need lineage from the original source to the write proposal, stored record, retrieval event, model context, and resulting tool decision. Microsoft describes this as end-to-end traceability; the same requirement should shape independent implementations.

Recovery must be designed before an incident. Deleting one record may not remove summaries, embeddings, cached plans, or downstream memories derived from it. Test correction and deletion as distributed invalidation problems, then verify that the agent stops recalling the poisoned state.

• Alert on unusual write volume, repeated rejected writes, cross-context retrieval, and memory-driven sensitive actions.
• Provide user and administrator views that explain what was remembered, why, from which source, and where it was used.
• Support quarantine, version history, scoped deletion, expiry, and restoration without requiring direct database edits.
• Track derived records so revoking one source invalidates summaries and memories built from it.
• Retain enough redacted evidence to reproduce the failure without preserving secrets or attacker payloads indefinitely.

Turn memory tests into release gates

OWASP recommends structured agent testing before production and after material changes to prompts, tools, memory, retrieval, policies, or model providers. Memory tests should therefore run as versioned regression cases, not as a one-time red-team exercise.

Keep the model version, memory policy, retrieval configuration, tool set, identity scope, test input, expected denial, and observed evidence together. Block a release when a known poison can persist, cross a boundary, or activate a sensitive tool without a current authorization decision.

This is where continuous offensive testing earns its place. The goal is not to claim that memory poisoning has been solved. It is to keep the write, retrieval, action, and recovery controls measurable as the agent changes. Persistent systems require persistent security evidence.