AI Agent Memory: When Does a Custom Agent Need It?
November 9, 2026 · Muhammad Hammad · Co-founder & Engineer, Vision Nexera · 8 min read
A custom agent needs AI agent memory only once it must learn things its data and tools cannot tell it. Start with stored transcripts and retrieval over them, add a small set of notes the agent writes, and run a scheduled job that checks those notes against the logs. Adopt a heavier memory product only when an evaluation on your own data shows it beats that baseline.
Agent memory had a loud fortnight. On 7 November Anthropic published The components of agent memory, which reached the Hacker News front page with 179 points. A day later, LoreKit's case that agent memory needs its OpenTelemetry moment drew 177 points by counting four incompatible memory formats on a typical developer machine. In the same week, Redis made Agent Memory generally available in Redis Iris and MongoDB shipped durable session and memory storage for Google ADK.
The research points the other way. A University of Illinois Urbana-Champaign team tested 11 memory systems against plain chunk retrieval in Memory Tax: Dissecting the Net Value of Agent Memory, published on 25 October. Across three benchmarks and three model backbones, by our count of the paper's Table 1, the memory system scored below a two-round RAG baseline in 90 of 99 cells.
Both signals are right. Memory helps when it stores the right things and stays correct. It hurts when it rewrites or buries evidence the agent could have retrieved directly. This article is about building the first kind.
Memory and retrieval solve different problems
Anthropic defines agent memory as knowledge an agent gains from experience and keeps between sessions, saved as context it can read later. That excludes changes to model weights, learning inside a single conversation, and compaction. In practice, a custom agent has four layers to choose from.
| Layer | What it holds | Typical tools | Earns its place when |
|---|---|---|---|
| Session state | The current run's messages and variables | Google ADK sessions, LangGraph checkpointers | Always |
| Retrieval over transcripts | Past runs, searched on demand | Postgres with pgvector, MongoDB Atlas Search | The agent must recall what happened |
| Written memory | Notes the agent writes for its future self | Anthropic memory tool, LangGraph BaseStore, Redis Agent Memory | The agent must learn rules no data source states |
| Injected context | A short file loaded into every session | CLAUDE.md, AGENTS.md | A rule matters on almost every run |
The Memory Tax paper explains why the upper layers can lose. Every memory step moves away from the raw history: it reorganizes facts, distills preferences, summarizes. Some of those changes help. Others drop, bury or rewrite the evidence a plain retriever would have found. The authors call the gains memory benefit and the losses memory tax. Net value is the difference, and in most of their tests it was negative.
If your agent mainly answers from documents that people wrote, you have a retrieval problem, not a memory problem. Our RAG vs fine-tuning framework covers that decision.
Build AI agent memory from the bottom up
Read the diagram from the bottom. Layer 1 is raw transcripts and run logs. Layer 2 is retrieval over those transcripts. Layer 3 holds written memory notes. Layer 4 is a small injected index. A scheduled consolidation job reads the transcripts and promotes checked notes upward. Build in that order:
- Log every run: prompts, tool calls and outcomes, each with a run ID. This is the ground truth for everything above it.
- Make the logs searchable. Hybrid search over past runs is the baseline every memory layer has to beat.
- Give the agent a small notes store for rules it verified, traps it hit and decisions people made.
- Add a consolidation job that checks notes against transcripts and merges, corrects or prunes them.
- Promote only what matters on almost every run into a short injected index.
Store what the system of record cannot tell the agent
Anthropic's Pokémon test is the clearest illustration. About 10,000 steps in, Claude Sonnet 3.7 had written 413 memory files across 225 folders and earned one badge. Claude Fable 5 had written nine files in two folders plus a single global.md dashboard, and earned four. Sonnet 3.7 saved facts it could read off the screen every turn, so its memory went stale and it spent turns tidying files instead of playing.
Business agents fail the same way. Do not copy CRM fields, order status or account balances into memory. Read them from the system that owns them. Store what exists only because the agent worked it out: a supplier portal that rejects invoices dated on weekends, the approval a finance lead always asks for, a query pattern that times out.
Say what memory is for, not what to write
Anthropic found that telling strong models a memory folder exists was enough. Adding a fixed rule such as "record only failures" removed much of the gain for Claude Opus 5 and Opus 4.7. Older models behaved differently: prompting Claude Sonnet 4.6 to read its memory before answering, and to test its theories, nearly doubled its score on the CL-Bench task.
The right memory instructions depend on the model. That is one more reason to rerun your evaluation every time you change models.
Schedule a job that checks memory against the logs
Agents write memory based on what looks important in the moment, so some notes start wrong and others go stale. Anthropic's answer is a scheduled offline pass, which it calls dreaming, that rereads transcripts and corrects notes. In one Pokémon run, a wrong note about a building's location sent Claude off a one-way ledge three times. After the pass corrected the note against the transcript, Claude went straight back into the cave and reached the next city.
The same consolidation pass can build structure. On a corpus of roughly 350,000 internal files, Claude with search plus a self-built wiki scored 61%, against 53% for search alone, and used about 20,000 fewer tokens per question. The wiki on its own scored 48%, below search alone. On merger questions that needed several documents, the wiki lifted scores from 73% to 87%. On single-document tasks it added nothing, and one wiki grew larger than the documents it summarized.
In a production stack this is an ordinary scheduled job: an n8n workflow or a k3s CronJob runs a consolidation agent with read access to transcripts and write access to the notes store. Have it produce a diff rather than a silent update, so a person approves anything that reaches the top layer. Keep that layer small. Claude Code, for example, loads only the first 200 lines or 25 KB of its MEMORY.md file into each session and leaves the rest to search.
Treat memory as untrusted input
Anything an agent reads can steer it, and shared memory widens who can write to it. LoreKit's post says it plainly: no format makes memory poisoning impossible. Design for it:
- Provenance: record who wrote each note (a person, the agent or an extraction service) and link it to the run that produced it.
- Scope: address every note to a tenant, user or project, so one client's lesson never reaches another client's session.
- Expiry: give operational notes a review date. Forgetting by default is safer than keeping everything.
- Validation: parse and check notes before they enter context, the same way you check tool output.
- Review: require approval before a note is promoted to the injected layer.
Our natural-language-to-MongoDB MCP server follows the same discipline: it treats the model's output as untrusted input and blocks destructive operations at a layer the prompt cannot reach. Memory deserves that treatment because it outlives the session that wrote it. Provenance is the control teams skip most often, and the one they miss first during an incident.
Keep memory in your own database with a plain envelope (scope, key, author, created, expires) rather than inside one vendor's format. LoreKit's comparison lists GitHub Copilot Memory, Letta, Mem0, LangGraph BaseStore and Graphiti among others, each with its own unit of memory and its own rules for forgetting. An envelope you own turns a later vendor switch into a migration script instead of a rewrite. Our LangGraph vs CrewAI vs AutoGen guide makes the same argument for orchestration code.
FAQ
What is AI agent memory?
AI agent memory is knowledge an agent gains from its own work and keeps between sessions, stored as notes or records it can read later. It differs from session state, which lasts one run, and from RAG over documents, which reads knowledge people wrote. Most production agents use all three.
Is a memory layer better than RAG for agents?
Not by default. In the Memory Tax study, most dedicated memory systems scored below plain chunk retrieval over the raw conversation history, often at higher cost. Memory did win on some queries that needed facts combined or restructured. Test both on your own data before you choose.
Which memory options work with LangGraph, Google ADK or n8n?
LangGraph ships BaseStore for namespaced long-term memory and checkpointers for session state. Google ADK separates session, state and memory services, with integrations from MongoDB, Mem0 and others. In n8n, a Postgres or Redis node plus a retrieval step covers most needs without a dedicated memory product.
How much does agent memory cost to run?
It depends on how much work happens when memory is written. Systems that extract facts or build graphs call a model on every ingest, so they cost more than storing transcripts and searching them. Track tokens per task and offline build time next to accuracy. Our AI agent development cost guide covers run costs in detail.
Run the baseline test before you buy memory
Take 50–100 real tasks from your agent's history and run them three ways: no memory, retrieval over transcripts, and the memory product or design you are considering. Score accuracy, tokens per task, latency and the offline cost of building the memory. Keep the winner, and rerun the comparison whenever you change models.
This is how we approach memory in the AI agents we build: transcripts first, retrieval as the baseline, written memory only where an evaluation shows it pays, and a human gate on anything promoted to every session. When the job is answering from your documents rather than learning from the agent's own runs, the work belongs in RAG and LLM applications.
The agents that remember best do not store the most. Their notes are few, and something, a person or a scheduled job, has checked each one against what actually happened.