Manuscript status
The supplied paper is labeled “Anonymous ACL submission.” It does not name its authors or establish acceptance or peer-review status. The original 13-page PDF is in English.
Study overview
The study compares Hindsight, Mem0, Honcho, OpenViking and Graphiti with hybrid RAG, full-context and no-memory baselines. It uses 1,000 questions derived from 200 conversations across LongMemEval, LoCoMo, Multi-Session Chat and Enron, grouped into 139 memory spaces. The model, embedder, answer prompt and judge are held constant while unrelated history increases to 0%, 50% and 90% of ingested tokens.
Main findings
- At an 8K-token context budget and 90% noise, hybrid RAG scores 96.8%, compared with 89.9% for Hindsight, the highest-scoring memory engine in this setting.
- For RAG and the four engines evaluated at all noise levels, the answer model reads available evidence correctly 94.5–98.8% of the time. Most errors occur when retrieval fails to deliver the evidence.
- Returning relevance-ranked text that fills the context budget is more robust in these runs than returning a fixed number of items or reserving space for recent memories.
- The reported 90%-noise RAG run costs $1.70. Building memory accounts for 86–98% of the total run cost for the four extraction engines tested at that level; write cost follows the number of sessions rather than token count alone.
How to interpret the results
These results describe this benchmark, not a general ranking of memory engines. The setup uses one seed, one model configuration and engine defaults without tuning. Questions and some noise were generated by models without a human audit, and the noise is non-adversarial. Honcho’s 90%-noise result is partly confounded by unfinished memory building at the three-hour settle limit. Graphiti was not evaluated at 90% noise. Benefits such as compact, inspectable and editable memory were not measured.
Practical implication
For question answering over growing conversation histories, the paper argues for checking whether retrieval actually delivers the relevant evidence, how much context the system returns, and whether memory construction has finished before evaluation. Accuracy and cost should be measured in the intended workflow rather than inferred from a clean-history benchmark.