My RAG had all the evidence. It still got 167 answers wrong.
retained text
I think we’re underestimating one of the biggest problems with AI agents: finding the right information doesn’t necessarily mean the agent can use it correctly. And I ran into this while researching temporal RAG. The problem Imagine asking an AI agent to compare two companies’ revenue across two reporting periods. It needs four specific facts. Not four relevant documents. Four facts, each tied to the correct company, metric and period. Now imagine those figures have been revised, or the corpus contains multiple versions of the same report. Your retriever can return genuinely relevant passages, citations can look correct, and the agent can still mix quarters, miss a required value or generate a confidently wrong comparison. The problem isn’t always hallucination. Sometimes the evidence is there, but the system doesn’t preserve what each piece of evidence is supposed to answer. What I built I developed ChronoRAG-G to investigate this, using an approach called OETRA (Obligation-Explicit Temporal Retrieval and Allocation). The central idea is fairly simple: instead of retrieving a pile of context for the whole question, identify exactly which facts must be established before generating the answer. The approach 1. Break the question into explicit evidence obligations, each representing one required fact. 2. Retrieve evidence for each obligation separately, rather than relying entirely on a pooled top-k search. 3. Preserve entity identity, temporal relationships, source provenance and competing evidence in a graph-structured context corpus. 4. Check whether the evidence actually matches the required entity, metric and reporting period. 5. Allocate compatible evidence to its corresponding answer slot before structured generation, with a bounded correction step. The graph organizes temporal evidence. It isn’t just unrestricted graph traversal replacing vector search; targeted dense retrieval still handles evidence acquisition and expansion. The results I evaluated it on 1,005 temporal QA questions over 480 updated earnings-call transcripts. \* 80.70% audited answer accuracy \* 95.92% strict trace-derived slot coverage \* 98.66% required-slot micro coverage But here’s the finding that interested me most: 167 incorrect answers still had complete evidence traces. The system had resolved the required evidence, yet answer construction still failed. That means improving retrieval alone won’t necessarily solve the remaining problem. Evidence completeness and correct evidence usage are separate things. Why I think this matters beyond RAG Think about financial research agents, compliance assistants, evolving company policies, regulatory histories or enterprise workflows where information changes over time. An agent acting on an outdated policy or mixing figures from different reporting periods isn’t just giving a bad answer. It could influence a real business decision. Those are potential applications, not validated deployments. My published evaluation is specifically on financial temporal QA. The research is public, and I’ve built an interactive demo showing the evaluation traces, evidence allocation and failure cases. I’m interested in hearing from people building production agents or enterprise RAG systems. When your agent gives a wrong answer despite retrieving the necessary information, how are you detecting that? Are you separately evaluating evidence completeness and final answer correctness, or just scoring the final output?
document fetched 2026-11-10 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xrl7l5/my_rag_had_all_the_evidence_it_still_got_167