# I benchmarked classic vector RAG vs Google's new OKF format vs both combined — same corpus, same 7 questions, all local (Ollama + ChromaDB) — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/d1bbb6f87ef830de

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-08-03 06:46 ingested 2026-08-03 09:25

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1ve5r8y/i_benchmarked_classic_vector_rag_vs_googles_new/)

## retained text



```
<!-- SC_OFF --><div class="md"><p>Google Cloud published OKF (Open Knowledge Format) on June 12th — a spec for storing curated knowledge as a directory of markdown files with YAML frontmatter. One concept per file, linked to each other, with an <a href="http://index.md">index.md</a> for progressive disclosure. The only required field is `type`.</p> <p>I wanted to know whether it actually fixes anything, so I built a test corpus and measured. Everything runs locally: qwen3:8b + nomic-embed-text + ChromaDB, no external APIs.</p> <p>SETUP</p> <p>- Corpus: 60 markdown files of fake-but-realistic company docs (wiki, table schemas, ADRs, 40 support tickets). 85 chunks at 800/100.</p> <p>- OKF bundle: 9 curated concepts covering the same ground.</p> <p>- 7 questions, each designed to trigger a different retrieval failure mode.</p> <p>RESULTS (7 questions)</p> <p>RAG OKF OKF+RAG</p> <p>correct 2 3 4</p> <p>tokens 6341 8625 8435</p> <p>Nothing passes. The combined layer gets twice what plain RAG does, at ~33% more tokens.</p> <p>THE ONE THAT SURPRISED ME</p> <p>Question: &quot;how do we calculate revenue?&quot; The corpus has a 2023 doc (deprecated, verbose, 4000 chars) and the current 2026 spec (terse, 500 chars).</p> <p>The deprecated doc splits into 7 chunks, the current one into 1. Three of the top-5 retrieved chunks came from the deprecated doc. The correct document ranked <strong>15th out of 85</strong> — behind a glossary, a customer table schema, and a support ticket about shipping costs to the Canary Islands.</p> <p>Raising k to 15 doesn't help: you'd pull in 6 chunks saying the wrong thing against 1 saying the right thing. A reranker can't fix it either — there's nothing in the chunk text indicating which is current. The date isn't in the chunk.</p> <p>OTHER FAILURE MODES THAT FIRED</p> <p>- Chunker split an 18-column schema table. The right file WAS in context; the table wasn't. Model said &quot;I don't know&quot; at both k=3 and k=5.</p> <p>- Composition: a metric definition needs 3 rules living in 3 separate files. RAG retrieved 2 of 3 and answered confidently, citing sources, never hinting anything might be missing.</p> <p>- Interesting pattern: it said &quot;I don't know&quot; when it had almost nothing, and said nothing when it had almost everything. It goes quiet exactly when it's most expensive.</p> <p>WHERE OKF LOSES</p> <p>Long-tail questions. &quot;Was there an incident with duplicate orders in March?&quot; — plain RAG nailed it over 40 messy, unreviewed tickets. Curating those by hand would be absurd. OKF alone failed it.</p> <p>TERMINOLOGY CAVEAT</p> <p>I'm using &quot;RAG&quot; as shorthand for the classic vector implementation. Strictly, an agent navigating an OKF index is also a RAG pipeline — just with structured retrieval instead of vector retrieval. The precise framing is &quot;classic vector RAG vs structured retrieval over OKF&quot;. Someone rightly called me out on this.</p> <p>Full code, corpus, bundle and the raw results.txt: <a href="https://github.com/JoaquinRuiz/rag-vs-okf">https://github.com/JoaquinRuiz/rag-vs-okf</a> git clone + uv sync and you can reproduce it. Curious whether anyone gets different numbers with a bigger model — question 4 was unstable across runs for me.</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/jokiruiz"> /u/jokiruiz </a> <br /> <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1ve5r8y/i_benchmarked_classic_vector_rag_vs_googles_new/">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1ve5r8y/i_benchmarked_classic_vector_rag_vs_googles_new/">[comments]</a></span>
```



document fetched 2026-08-03 09:25 · canonical https://reddit.com/r/LocalLLaMA/comments/1ve5r8y/i_benchmarked_classic_vector_rag_vs_googles_new
