“Memory is the mother of all wisdom.” — Aeschylus (525–426 BC)
In humans, memory underpins learning, identity, and reasoning. In LLM agents, it is mostly missing. This series is about how modern agents can remember, guided by how brains do.
Table of Content
What’s Wrong with Current Agents
Anyone who has used a coding agent knows this screen:
Imagine you are halfway through a refactor. The agent has finally grasped your module layout, your naming conventions, and what you actually asked it to build, and everything looks ready to run. Then the session limit hits, and the work stops where it stands. You wait, the credits refill, and you pick up again, but the whole history has to ride along in the context window, so every turn costs more tokens than the last, and you march straight back toward the same limit. The wall is not really the credit cap; it is that the context keeps growing and you keep paying to carry it.
👀 This is not a quota problem. This is a memory problem.
Memory Is an Economic Necessity
Modern LLMs rely on attention to attend to past information. Unfortunately, naive attention without memory is quadratically expensive:
Because of the attention mechanism, LLMs need to attend to the whole context every forward pass. As a result, LLM agents re-pay for the same context every turn and every session:
Full history re-sent each turn. Token cost grows with conversation length.
Long prompts. Attention is quadratic, so latency and compute per call rise with length.
Facts re-derived, retrieval re-run, dead ends re-explored. The agent pays again for work it already did.
👀 Besides cost, another issue is that even the top expensive agents are not that good.
On the DeepSWE benchmark, the top agents score only 50–70% pass@1 at roughly $6–18 per task and 20–45 minutes each. The failure modes are missed requirements, regressions, unverified assumptions, and knowledge gaps, which represent memory failures rather than reasoning failures.
Current LLM agents are capable but forgetful, and the same gap shows up wherever they run for long. The consequences include:
A coding agent re-learns your stack every session.
An assistant forgets a preference you mentioned yesterday.
A multi-day task loses its own thread partway through.
None of this matters much for a single question and answer, but as agents shift from single-turn tools into long-lived collaborators, forgetting stops being a nuisance and becomes the bottleneck.
👀 Agents work well inside a context window and badly outside it.
The Usual Fixes Aren’t Enough
The bottleneck is well known, and the field has three standard solutions for it. Sometimes they help, but many times they fail.
❌ Longer context. Extending the window helps, but it does not fix the underlying problem. Once the window fills, the oldest turns still fall off the edge, and the cost keeps climbing as the conversation grows. Worse, models do not attend evenly across a long prompt: they use the middle far less reliably than the beginning or the end, a failure well documented as the “lost in the middle” effect [2]. Don't be fooled when a model advertises a context window of a million tokens or more. A large window is not the same as a reliable memory. As the prompt grows, the model attends to it less and less evenly, and information buried in the middle is effectively half-forgotten even though it is technically still "in context."
❌ Flat RAG. Retrieval-augmented generation adds an external store, but a naive version embeds each passage in isolation, so nothing ever integrates across separate experiences. Because retrieval works by one-hop similarity, it returns passages that look like the query rather than the chain of facts an answer actually requires, and multi-step questions fall straight through the gap.
❌ Fine-tuning. Distilling knowledge into the weights is slow and expensive, and it cannot adapt to an individual user or update on the fly as new information arrives. On top of that, each new round of training tends to overwrite what came before, so the model still forgets catastrophically.
👉 None of these provides a principled memory solution. Each patches one problem, either context length, lookup, or weight updates. A real memory solution should clearly define a holistic memory architecture and pipeline that efficiently support long-term retrieval. To achieve this, it helps to look at the one system that already has principled memory: humans brain.
Four Processes From the Brain
The brain converts experience into memory through four main processes. Each maps to something an agent can do.
🟡 Encode. The hippocampus binds an experience into a trace. Example agent: write a record of what happened.
🔴 Consolidate. Sleep replays and integrates memories into the neocortex. Example agent: merge, link, and compress over time.
🟢 Retrieve. A partial cue reactivates the whole memory. Example agent: search for the right trace by relevance.
⚪ Forget. Synaptic decay clears interference. Example agent: prune to stay bounded and coherent.
A fifth process governs the other four: control, which decides when each one fires. We will refer to them as 4+1 processes.
Every system in this series implements some subset of these five processes, and the sharpest question you can ask about any memory architecture is which of them it quietly leaves out. But knowing how memory operates is only half the picture. Before we can build these processes, we need to know what memory actually holds.
Three Kinds of Long-Term Memory
Cognitive science divides long-term memory into three systems, and it maps cleanly onto what an agent must store:
Tulving [1972] split long-term memory into episodic, seated in the hippocampus, and semantic, seated in the neocortex, and this systems view is now standard in cognitive neuroscience [3]. Alongside the two long-term stores sits a small, fast working memory: the active, short-term context you are thinking with right now. In an LLM, working memory is simply the prompt, i.e., whatever currently fits in the context window.
Seen through this lens, flat RAG turns out to implement only one system, and it implements it awkwardly. The content it holds is episodic, a pile of specific passages, yet it queries that content as though it were semantic, reaching for general facts by surface similarity.
👀 That mismatch is a major reason flat RAG so often fails in practice.
Knowing the types of memory content, though, still leaves one question open: how does the brain locate the right piece of information, in the right store, at the moment it is needed? The answer is a single mechanism, and it is the one that matters most for agent design.
Hippocampal Indexing
In the brain, the hippocampus does not keep full memories. Instead, it keeps a sparse index of pointers into patterns spread across the neocortex, and on recall it reactivates those patterns together.
Think of how a familiar smell can trigger a vivid, detailed memory. The smell carries almost none of the memory itself. It acts as a partial cue that points to the full trace, and the brain follows that pointer to reconstruct the whole scene. The idea can go straight into agent design, which raises the research question the rest of the series keeps returning to.
🧠 If memory is an index of pointers rather than a pile of documents, what should an agent’s index look like?
Fast Hippocampus, Slow Neocortex
The index explains recall, but it does not explain how a memory formed in a single second becomes knowledge that lasts for years. For that, the brain keeps two stores instead of one, an arrangement known as Complementary Learning Systems, and it comprises a fast learner beside a slow one.
🔵 Hippocampus — fast. It binds a new episode after a single exposure, and because it is sparse, it avoids interference between memories.
🔴 Neocortex — slow. It extracts structure across many episodes, and because it changes slowly, it stays stable; this is the seat of semantic memory.
Consolidation is the bridge between the two: it carries knowledge from the fast store into the slow one without overwriting what the slow store already holds. Writing something down is easy, but writing it down without destroying what was already there is exactly the trick that catastrophic forgetting cannot pull off.
Once you align this two-store design to modern agent design, it takes on a shape that should look familiar to anyone who has built with LLMs.
Hippocampal Memory Is RAG
Spens and Burgess [2024] set out to model how the brain’s two memory systems cooperate, and the model they arrived at lines up, point for point, with RAG [4].
Hippocampus = retriever. It stores events in compressed conceptual form — the gist, plus whatever details were surprising.
Neocortex = generator. It is an LLM-like network that decompresses those traces and reconstructs the full event.
Working memory = context window. The retrieved traces are loaded in as the prompt.
Consolidation = training. The generator is gradually trained on replayed memories until it can recall them without the trace at all.
That coincidence is good news, because it suggests RAG is a natural solution rather than a lucky hack. It is also a checklist of what plain RAG is missing, since the biological version comes with consolidation, forgetting, and gist-versus-detail compression that naive RAG mostly lacks.
The brain has now handed us a list of principles we can borrow for agent design: 4+1 processes, 3 kinds of content, an index for recall, and 2 stores that move knowledge from fast to slow. The engineering question is where inside an agent all of it should be organized.
Where Should Memory Live?
The first question to consider when designing agent memory is where the memory physically sits. There are two answers, and they define the two paradigms.
Paradigm 1: Inside the Model
In the first paradigm, memory is architectural. It lives in the model’s own parameters or state; it updates as the model runs, and it stays learned, compressed, and implicit. Within that paradigm, there are two places memory can settle.
In the state, memory is the KV-cache or an RNN’s hidden state. It is fast and accurate, but it is bounded and vanishes when the run ends, so it behaves like short-term working memory.
In the weights, memory is the parameters themselves, updated at test time. It is effectively unbounded and highly compressed, but it is lossy and opaque, so it behaves like long-term memory.
The appeal of this paradigm is that each of the earlier fixes reappears as a property it supplies for free. Longer context becomes compressing the past into parameters, flat RAG becomes memory that is learned instead of looked up, and fine-tuning becomes an update that happens online as the model runs. Put simply, a plain LLM learns to attend, whereas an LLM with in-model memory learns to remember.
🧠 Can a model learn to remember as it runs, without leaning on an external database?
Part 2 answers that in detail through Titans, which turns prediction error into a signal for what to store, and ProCL, which adds the consolidation step that online writing alone cannot provide.
In-model memory is powerful, but it is sealed: you cannot open the weights and read what the model has decided to remember about you. The second paradigm gives up some of that compression in exchange for exactly that transparency.
Paradigm 2: Outside the Model
In the second paradigm, memory is external. It sits in a store the agent reads from and writes to explicitly, and that separation buys four properties the in-model version cannot easily offer.
✅ It is inspectable, so you can read, edit, and audit whatever is stored.
✅ It is unbounded, so it grows without retraining the model or enlarging the context window.
✅ It is structured, holding graphs, trees, and documents rather than a flat heap of chunks or vectors.
✅ And it reframes the long-context problem, because the goal is no longer to squeeze the past into weights but to organize a store so the right item can be found.
This is where most of the commercial energy has gone. Market research values the agentic AI orchestration and memory systems market at $10.9B in 2026 and projects it will reach $210.5B by 2035, a 38.9% compound annual growth rate [5]. Memory layers and vector databases are the fastest-scaling piece of that stack, which is unsurprising once you note the same report's claim that more than 80% of new databases in 2026 were built and managed by AI agents rather than people. The money is following the same logic: Mem0 raised $24M across seed and Series A in October 2025 for model-agnostic memory infrastructure, and LangChain reached a $1.3B valuation after its own round the same month.
👀 The question now is no longer whether agents need external memory but rather how that memory should be organized.
🧠 Can external memory be organized well enough to support encoding, retrieval, consolidation, and forgetting all at once?
Part 3 takes up that question with HippoRAG and EM-LLM on the retrieval side, and Mem0, Zep, MemoryBank, Nemori, TiMem, and EverMemOS on the consolidation and forgetting side.
The Trade-Off
Set side by side, the two paradigms are not really rivals; they are two complementary options we can consider when designing agent memory.
A serious agent will want both, pairing a fast in-model store for the task in hand with a durable external store that survives across sessions. That combination is close to the arrangement the brain already runs.
The Unsolved Process: Control
Both paradigms hand the agent the machinery to carry out the four processes, yet neither of them decides when each process should actually fire. That decision is the fifth process, and it is the one that has been actively investigated recently.
The brain assigns the job to the prefrontal cortex, its central executive, which gates what is allowed into working memory and reallocates attention as the situation changes. LLMs have no such organ; their attention is a fixed, bottom-up filter with no one steering it from above. Agents paper over the gap with hardcoded heuristics, fixed rules, no learning, and no top-down adaptation, which works until the moment the situation stops matching the rules.
🧠 Can we build an adaptive, meta-level controller that decides when to remember?
Control is not only the least mature part of the field but also the worst measured. Benchmarks such as LoCoMo [6] and LongMemEval [7] score end-task accuracy over long dialogues, which folds retrieval, control, and generation into a single number and makes it impossible to see which one failed. What no benchmark yet isolates is the control decision itself: arbitrating between contradictory memories, suppressing interference, and reallocating focus the instant a goal shifts.
Part 4 closes the series on this thread, tracing the arc from hand-designed control in MemGPT, through RL-learned control in Memory-R1, to the schema-guided executives of MemoBrain and MemCoE, and then to the problems that remain open.
Roadmap
Part 1 — this post. Why memory is the bottleneck, the 4+1 processes worth borrowing from the brain, and the two paradigms that organize everything else.
Part 2 — Inside the model. Architectural memory in state and in weights, Titans and the idea of learning by surprise, why writing is not the same as consolidating, and ProCL’s take on continual LoRA.
Part 3 — Outside the model. Brain-inspired retrieval with HippoRAG and EM-LLM, then consolidation and forgetting across Mem0, Zep, MemoryBank, Nemori, TiMem, and EverMemOS.
Part 4 — The missing executive. Who decides when to remember: the arc from hand-designed control in MemGPT, through RL-learned control in Memory-R1, to the schema-guided executives of MemoBrain and MemCoE, and the open problems in measuring control.
Agents already have capable memory, but they lack the executive that decides when to encode, retrieve, consolidate, and forget. Closing that gap, and learning how to measure it, is the frontier.
References
[1] DeepSWE benchmark results, 2026. https://deepswe.datacurve.ai
[2] Liu, Nelson F., et al. “Lost in the middle: How language models use long contexts.” TACL 12 (2024): 157–173.
[3] Tulving, Endel. Episodic and Semantic Memory. Academic Press, 1972, pp. 381–403.
[4] Spens, Eleanor, and Neil Burgess. “A generative model of memory construction and consolidation.” bioRxiv (2024). doi:10.1101/2024.11.04.621950.
[5] Memory/orchestration market sizing and funding figures, 2025. https://dimensionmarketresearch.com/report/agentic-ai-orchestration-and-memory-systems-market
[6] Maharana, Adyasha, et al. “Evaluating very long-term conversational memory of LLM agents.” ACL (2024): 13851–13870.
[7] Wu, Di, et al. “LongMemEval: Benchmarking chat assistants on long-term interactive memory.” ICLR (2025).