No. RAG Cannot Replace a Good Model.
retained text
I embedded four policy sentences with nomic-embed-text on Ollama and scored each against one question: "Can I get my money back if I cancel my wedding?" "Cancellations made 30 days or more before the ceremony receive a full refund." 0.7216 "Cancellations made 30 days or more before the ceremony do not receive a refund." 0.7260 The no-refund sentence ranked first. Across five phrasings of the question, the two never scored more than about 0.02 apart, and scored against each other they come out at 0.97. Embeddings encode what a text is about. Whether it says yes or no is a detail they mostly lose. Then I asked whether I could bring my dog. None of the four sentences answers that. The top hit was the marriage license requirement at 0.59, because vector search has no concept of "no match." Something is always nearest. A distance cutoff handles the dog. Even a good one lets through the sentence that says the opposite of the truth, because it's about the same topic. The claim going around is that RAG makes the model a detail, or that it turns the vector store into the model's own weights. Insert a million rows and the weights file doesn't change by a byte. Retrieval decides what the model sees. The weights decide what it does with it: notice the two chunks contradict, work out that the 2026 policy supersedes the 2023 one that ranked first, and say it doesn't know about the dog instead of inventing a pet policy. Commercial RAG fails the same way. LexisNexis marketed Lexis+ AI with "hallucination-free" linked citations. A Stanford/Yale team ran 202 preregistered legal queries through it and Westlaw's AI-Assisted Research by hand. Lexis hallucinated on more than 1 in 6, Westlaw on 1 in 3, and for Westlaw a reasoning error contributed to 61% of the hallucinated answers. The retrieved results often looked relevant. I run a production agent, an AI wedding planner for a Las Vegas wedding chapel, on Claude Opus through Microsoft.Extensions.AI. Live data comes through a .NET 10 MCP connector I wrote against the booking system, because whether a Saturday in October is open is a WHERE clause, not a similarity question. The persona is a senior planner running a six-phase consultation, and nobody retrieves the discipline to follow six phases through a long conversation. When Anthropic's API overloads mid-stream, the agent falls back to Sonnet. The fallback is also a frontier model. Has anyone gotten a reranker to reliably separate contradictions like the refund pair, or does that always end up on the model? Write-up with the code to reproduce this (C#, Agent Framework's TextSearchProvider, SQL Server 2025 vectors, and the CU9 gotcha where a DiskANN index makes the table read-only) in the comments per this forum's rules
document fetched 2026-09-26 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1wpywtv/no_rag_cannot_replace_a_good_model