Got Gemma 4 12B to keep reliable long-term memory locally: saved sessions went from 52 to 94 of 96, and it kept more correct facts than Sonnet 5.5 on the same pipeline.
retained text
I've been building PCS, a free, open-source (GPL-3.0) Windows app for long-running conversations with an AI that actually remembers. It runs fully local on my 16 GB card, and the local route has no cloud fallback.
\*\*What runs on your PC\*\*
\- \*\*Conversation and memory on Gemma 4 12B QAT (Q4\_0)\*\* through a pinned llama.cpp CUDA build. Context is 128K by default and goes up to 256K, with an FP16 or Q8\_0 KV cache. On my RTX 5080: about 78 tok/s decode and 3,100 tok/s prefill at 8K-token prompts.
\- \*\*Long-term memory the model writes and checks itself,\*\* kept in an encrypted vault. Every memory links back to the passages it came from, and you can correct it or make it forget.
\- \*\*Memories can come to mind on their own.\*\* Every time you type or speak, PCS matches your words against the vault on your PC and hands the model up to three related memories or earlier turns, so it doesn't have to decide to go looking. No extra model call, and nothing when nothing is clearly related. (Experimental and off by default; it matches words, not meaning.)
\- \*\*It can also go looking itself.\*\* For anything that didn't come to mind, ask "what did we decide about the project?" and it runs a memory lookup on its own.
\- \*\*Long sessions keep going.\*\* At about 80K tokens it compacts the working context and keeps recent exchanges whole. Older wording stays searchable for the rest of the session.
\- \*\*Voice:\*\* offline English listening on CPU or GPU, and Kokoro or Windows voices. On my NVIDIA card, listening and Kokoro are ready in about 6 seconds (it used to take 32).
\- \*\*Vision:\*\* connect OBS and pick a scene. It only sends a frame when your question is about the screen.
\- \*\*Search by meaning on the CPU\*\* with a local Nomic model, so no cloud embeddings.
\- \*\*Your own GGUF models:\*\* a compatibility check (text, tools, structured output, vision), llama.cpp auto-fit with RAM offload, and up to 8 saved model profiles.
\- \*\*A map of your themes that runs without any model call.\*\* Try it in the browser: [https://pcs-personalcontinuitysystem.github.io/PCS-site/continuity-map.html](https://pcs-personalcontinuitysystem.github.io/PCS-site/continuity-map.html)
The same vault also works with Claude, ChatGPT, Gemini, OpenRouter and others if you ever want a cloud model. You never have to.
\*\*Making a 12B model trustworthy at memory\*\*
This was the hard part, so I tested it properly. I used scripted multi-session histories (plans that change, corrections, things that end), answer keys sealed before any run, and blind grading. The wins:
\- \*\*Saved sessions 52/96 → 94/96. Meanings kept 68% → 94%. Stale or forbidden claims 36 → 2.\*\* The fix was in the reviewer, not the writer. The reviewer kept accepting drafts while citing passages the writer hadn't used (mostly the assistant's own sign-offs, like "good luck with the move!"), and the grounding check then blocked the save. Restricting the reviewer to the writer's declared sources fixed it.
\- \*\*12 topics interleaved in one vault: sessions saved went from 28 to 75 of 82, and current facts kept from 23 to 81 of 96.\*\* The fix was to stop letting the model rewrite whole memories. It now writes short statements with sources ("added / changed / corrected"), and the code assembles the memory. Old statements get an explicit verdict, so ended plans move into history instead of staying current.
\- \*\*On the same pipeline, Gemma ended with more correct current facts than Claude Sonnet 5.5: 120–124 vs 96 of 135.\*\* To be fair to Sonnet: wherever it saved, it made 1 error against Gemma's 12–14. It lost on volume. Its own reviewer rejected 18 of 84 sessions (strict, and mostly right to be), and 4 of its replies put a sentence before the JSON. Grammar-constrained output makes that impossible locally.
\*\*Lessons if you're building something similar\*\*
\- \*\*Small, grammar-constrained calls that each ask one question\*\* ("has this ended?", "does this belong here?") beat tweaking one big prompt. They give the same answer run after run; the big writer call doesn't.
\- \*\*Record every request and replay it.\*\* A replay takes about 3 minutes per test instead of a 2-hour rerun. At temperature 0.3 with no seed, resend a packet 10–20 times before you believe a change.
\- \*\*Shorter isn't automatically better.\*\* Cutting my 4,600-token writer prompt to 1,600 tokens halved the saves (52 → 28 of 96).
\- \*\*Gemma 4 26B-A4B didn't help.\*\* It needed changes to fit on 16 GB, crashed after long prompts, and didn't reduce errors in my test.
\*\*Still open:\*\* Gemma's main memory error now is retiring facts that are still true. The test histories are fictional, there are only a handful of runs per version, and the answer keys were written by a separate AI agent. Windows x64 only.
Free, no subscription, about a 42 MB download (models are separate): [https://github.com/PCS-PersonalContinuitySystem/PCS](https://github.com/PCS-PersonalContinuitySystem/PCS)
Happy to go into any part of the pipeline. If you've tried long-term memory with a local model, what broke for you?
document fetched 2026-11-20 00:41 · canonical https://reddit.com/r/ollama/comments/1y0anyg/got_gemma_4_12b_to_keep_reliable_longterm_memory