Agent memory is not retrieval over your codebase

"We gave the agent memory" usually means an index was built over the repository. Chunk the files, embed them, retrieve the nearest chunks, put them in context.

That is a useful thing to have. It is also not memory, and the difference is not pedantic, because the two solve different problems and a team that has built one often believes it no longer needs the other.

The distinction

Retrieval answers: what does the code say?

Memory answers: what did we conclude about the code, and does that still hold?

The first question has an answer that is already written, in the repository, in a form a machine can read. Indexing makes finding it faster. Nothing is created, only located.

The second question has an answer that exists nowhere. It was produced by a person or an agent doing work, it lived in a session or a head, and unless something recorded it, it is gone. No index over the source can recover it, because it was never in the source.

An example, since this is easy to wave at abstractly. An agent investigates why a queue consumer occasionally processes a message twice. It reads the consumer, the broker configuration, the deployment manifest, and the retry policy. It concludes that acknowledgement happens after a database write that can exceed the visibility timeout under load, so the message is redelivered while the first attempt is still running.

Retrieval can return the consumer file. It can return the timeout value. It cannot return that conclusion, because the conclusion is a relationship between four files and a load condition, and it is not written in any of them.

What is missing from the source, always

Some categories of knowledge are structurally absent from a codebase, and no amount of indexing changes that.

Conclusions. As above. The finding is a claim about the system, derived from several places at once.

Rejected explanations. The investigation checked the broker config and the client library version first. Both were fine. That is genuinely valuable, because it is what stops the next investigation from spending its first twenty minutes there, and it is the single thing least likely to be written down anywhere.

Reasons. Why the timeout is thirty seconds. Someone chose it, for a reason, probably in a review comment. The code shows the value and is silent about the reason.

Non-obvious relationships. This service copies its workflow file from that one. Those two constants have to stay in sync. Nothing in either file says so.

Validity. Whether any of the above is still true after the last four weeks of commits.

Retrieval over source gives you the what and none of the rest.

Why this matters more with agents than with search

Human developers compensate. They read the retrieved chunk, notice it does not quite fit, and ask a colleague. Retrieval is one input among several, and the person integrates.

An agent given a set of retrieved chunks treats them as the available evidence and reasons from them. If the chunks are the source and the source does not contain the conclusion, the agent will derive the conclusion again from scratch, which is the expensive outcome, or derive a plausible wrong one, which is worse. Either way the retrieval step did not save it anything except file discovery.

So the practical effect of calling an index "memory" is that a team stops looking for a memory. The gap stays exactly where it was, now behind a feature that appears to have covered it.

What memory has to add

Four things, none of which an index over source provides.

Capture. Something has to record the conclusion when it is reached. This is the step that fails in every manual system, because it asks a person to stop and write at the moment they most want to move on.

Evidence. A conclusion without its basis cannot be checked. "Acknowledgement happens after the write" is a claim; the four files and the specific lines are the evidence, and keeping them attached is what makes the claim auditable later rather than merely repeated.

Reconciliation. The claim has to be re-checked against the repository as it moves. This is where a claim store differs most from a document store: the store has to be able to say "the file this rests on changed after this was recorded" without a human prompting it. See context rot.

Scope. Team wide and across the repositories the team owns, because the conclusions worth keeping are usually about the seams between services rather than the inside of one.

The two together

This is not an argument against indexing. Retrieval over source is the right tool for finding code, and any memory system needs it, because the evidence a claim points to has to be located and read to be reconciled.

The argument is that they compose rather than substitute. Retrieval finds the material. Memory holds what was concluded from it and whether that conclusion survives contact with the current code. A system that only does the first will let a team re-derive the same conclusions indefinitely, quickly and at scale, which is what most teams using agents are doing today.

REM is being built as the second layer, with the first as an input rather than a competitor: evidence is immutable, claims are versioned, truth is reconciled against git, tests and history, and what gets compiled into an agent's context is the surviving claim rather than the raw chunk. It is in private beta.