glossary//agent-memory
agent memory
Agent memory is an AI agent's capacity to retain and retrieve information from prior turns and sessions.
The term names a capability, not an architecture. An agent has memory when it can answer a question that depends on something from three sessions ago, and lacks it when every conversation begins from zero. That is the whole definition, and it says nothing about how the retention works.
The nearest neighbour is the memory layer, which is the component that implements the capability: a specific store with its own shape and lifetime, sitting inside the harness. A system usually runs several at once. Agent memory is the what and the why; a memory layer is the how. You can say a system has agent memory without knowing whether it keeps a log, a digest table, an entity graph, or all three.
What it looks like in a real system
Memory is easy, retrieval is hard is the case study. Rheo, an internal Claude assistant running on claude -p behind a Flask and Telegram webhook, had conversation history, search tools and a memory API in its first version, and still forgot things constantly. Everything was stored. Nothing came back at the right moment. The essay's closing line is the point: memory is easy, retrieval is hard. Storage was never the hard part. Getting the right slice in front of the model, at the moment it matters, is.
The design that fixed it uses three layers with different lifetimes: a conversation ledger that keeps everything forever, session digests for recent episodic recall, and memory items that hold durable facts until something supersedes them, with boot-time injection tying them together.
Two ways people build it
Structured memory graph workers stakes out the split. Most agent memory today is a vector store with a fuzzy retrieval layer: embed the facts, search by cosine similarity, tune a threshold, hope the right thing comes back. Mem0 and Zep both work this way, and for plenty of use cases that is the right call. Recallatron bets the other way, on an entity graph, episodic timelines, procedural notes and structured query, where the graph and timelines are the primary structure and retrieval works by name and relation, not vector similarity; any embeddings are a derived, optional index on top, never the store of record. Ask it about a person and it returns the node, not the five nearest neighbours in vector space.
Give the model the tools, not the context cuts it a third way, by query mode rather than by storage: episodic, what happened and when; semantic, what is true about the world; procedural, how the user likes things done. But the three are not separate at write time. A single turn can drop a record into all three at once.