← sources

I tested 12 AI memory systems across 1,800 tasks. A plain Markdown wiki still tied for first.

source: reddit (reddit:AI_Agents)published 2026-09-18 13:24ingested 2026-09-19 02:37

retained text

Last month, I shared my test of 8 AI memory systems here and it got over 100,000 views. After 1,800 more tests, a plain Markdown wiki still scores the highest, but this time one tool matched its score.

**The setup**

We used Claude Code agents running Opus 5 with each of the 12 memory systems. Each agent got the same work across multiple sessions and 150 questions.

Each memory tool runs through weeks of simulated work. Each agent chooses what to store and remember, guided by each tool’s own documentation.

**The results**

Cognee and Karpathy’s Wiki tied for the highest accuracy score at 97.1 out of 100.

Using Cognee is significantly cheaper, at about half the wiki’s token cost at $256 per 1,000 questions vs. $503.

Every memory system has tradeoffs though, Cognee was the slowest system we tested, taking over five times as long per question as the wiki.

Across the tests, the biggest reason for failure was agents declining to answer questions they should have been able to answer. That accounted for 61% of failures. These failures can happen in three ways:

1. The agent never saved the information, so there was nothing to retrieve.
2. The agent never used its memory tool before declining to answer.
3. The agent searched its memory one or more times but gave up and said the information wasn’t there.

Sometimes declining to answer is the correct response. Our False Memory Check asks about facts the agent was never given. Every system aced this test except gbrain, which made up an answer when tested.

We also tested whether each system leaked information it had been told to forget. This matters when users ask an agent to delete personal or confidential information. 9/12 systems never repeated the facts they’d been told to forget, while three leaked data that should have been forgotten.

**If I were choosing today**

If you’re an indie hacker and don’t need to share memory with others, I’d stay local with Karpathy’s wiki or Mnemosyne. If token cost isn’t a concern, I’d use the wiki. If it matters, I’d use Mnemosyne, which scored 95.7 out of 100 at about $302 per 1,000 questions.

For shared memory across a team, I’d look at Cognee or Mem0. Cognee tied for the highest accuracy, but Mem0 was close behind and answered in less than a third of the time.

If the lowest token cost is your priority, Hindsight was the cheapest in this test at about $233 per 1,000 questions.

The full rankings, accuracy scores, failure breakdowns, speed and cost comparisons are available for free in the first comment.

Next we’re benchmarking the built-in memory of popular agent frameworks such as Hermes, OpenClaw and Claude Code, comparing them with each other and with agents using these memory tools.

document fetched 2026-09-19 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1wjq4vh/i_tested_12_ai_memory_systems_across_1800_tasks_a