Engineer Pin Lin (pinglin.tw) published a controlled comparison of three 'shapes' of AI agent memory: files curated by the model itself, structured stores that automatically extract atomic facts, and 'experience' trained into the model's weights via reinforcement learning. The same model, Qwen3.6-35B-A3B-mxfp4, and a single agent loop answered questions on two long-term dialogue memory benchmarks, with only the memory layer changing: on held-out LongMemEval-S, the structured store beat files by 28.7 points while spending roughly fifteen times fewer chat tokens, though files more honestly admit when they don't have the correct answer. The practical value of the work, however, is broader than an architecture race: scattered vendor numbers turned out to be incomparable, and the author proposes a verifiable methodology — line-by-line verdicts with an automatic verifier that anyone can recalculate.

image
image

What happened

Pin Lin built an experiment where, with a fixed model Qwen3.6-35B-A3B-mxfp4 and a single agent loop, only the memory layer changed, and answers were checked on two long-term dialogue memory benchmarks, LongMemEval and LoCoMo. The first form — files that the model maintains and curates itself; this approach is used by Claude Code, Cline, Cursor, and Windsurf, and the author calls OpenClaw the most developed implementation. The second — a structured store that automatically extracts atomic facts from dialogues; this is how mem0, Letta/MemGPT, and Zep work. The third — 'experience' written into the model's weights: the MemHarness system trains an observe-retrieve-critique-reconstruct-act policy using GRPO on a 7B model. On held-out LongMemEval-S, the store scored 0.7361 versus 0.4491 for files, while using 19.3k chat tokens per question versus 286.5k; on LoCoMo, the store's lead reaches 38 points. The only metric where files won was recognizing questions without a correct answer: 0.889 versus 0.778 for the store. All numbers and line-by-line verdicts are published in the a40-labs/memory repository along with a script that recalculates every result.

Context

Agent memory has long been a separate product: vendors sell it as a layer to place on top of any model, and choice has still relied on single numbers from benchmarks like LoCoMo. The research shows these numbers are incomparable: the same LoCoMo result for Zep appeared as 84, 58.44, and 75.14 in different materials due to different methodologies for counting the adversarial category, and mem0 was cited as both 67 and 92.5. A hidden variable turned out to be the evaluation stack: replacing the 'reader+judge' pair from a local 35B to gpt-4o-mini shifts the result by 6.9 points — more than the gap between any of the compared stores, so numbers from different pipelines cannot be directly recalculated into each other. Against this backdrop, a controlled protocol where only the memory layer changes and every number is accompanied by a line-by-line verdict is a rare attempt in this field to make measurement reproducible.

Why this matters for the industry

For teams building agents, the main consequence is that the memory layer can now be changed independently of the model and agent loop, and its contribution measured, rather than taking marketing slogans at face value. The 'files vs store' choice becomes a measured trade-off between accuracy, token cost, and honest 'I don't know' refusal. A second signal is even more significant: hidden infrastructure affects metrics more than the architecture itself — hosted Zep Cloud beat the same vendor's open-source engine Graphiti OSS by roughly 21 points with the same judge, and the hidden ingest pipeline of the hosted service is the prime suspect here. The conclusion on applicability conditions sets investment boundaries: on WebShop, memory gave +4.2 for a weak model and zero for a frontier model, meaning it only helps where the actor has room to grow — a direct argument for teams deciding whether to invest in memory on top of a strong model. The work also brings rare honest negative results that save others' budgets: the author's own hybrid is statistically indistinguishable from a simple vector index (+0.3 points), and 'dreaming' consolidation and an associative graph in the measured configuration had zero effect. If this practice becomes established, vendors will have to publish line-by-line verdicts with a verifier and disclose ingest pipelines, and choosing a memory provider will become a checklist of 'architecture plus ingest plus judge.'

Why this matters for users

For those building agents themselves, the research yields a simple order of operations. File memory — a MEMORY.md index plus grep on topical files — is cheap, requires no infrastructure, and honestly answers 'I don't know' when the answer isn't in the records, but on long dialogue histories it loses to the auto-extracting store on both accuracy and token cost. A rough rule: for a cheap personal agent, use files; for a product with long user history and a paid tier, use a structured store. A separate warning for enthusiasts experimenting with RL-trained models: crudely connecting an episode bank to such a model worsened the result from 76.4 to 70.1 on raw replay, meaning putting experience in context doesn't yet mean making the model use it. All conclusions can be verified without installing dependencies: after git clone of the a40-labs/memory repository and running python3 scripts/verify_all.py, every number is recalculated, and a toy-chatbot with file memory that works with any OpenAI-compatible endpoint is nearby.

What is still unknown / limitations

The evidence base for the comparison is one base model in mxfp4 quantized format, two dialogue benchmarks (LongMemEval and LoCoMo) plus WebShop, so the transfer of the store's lead and 15x token savings to other models, domains, and quantization regimes should not be drawn from these data. The conclusion that 'memory only helps where the actor has room to grow' currently relies on WebShop and a limited set of models — until verified on frontier systems, it is a hypothesis. The explanation of the gap between Zep Cloud and Graphiti OSS through a hidden ingest pipeline is an interpretation, not a measured fact, and the work should be viewed not as a 'choose in an evening' recipe, but as a reason to run the verifier on your own data.

Sources

Author

Look at AI, editorial team