An unanswered RAG question can mean three very different things
retained text
I've been thinking about how to evaluate a RAG system when the right response isn't a confident answer. Take a simple question: can this order be reshipped? 1. Nothing useful was retrieved. Maybe the right record exists and search missed it. 2. Relevant evidence was retrieved, but a required fact is missing. We have the order and the returns policy, but no confirmed delivery status. 3. The sources conflict. The carrier says delivered; the warehouse record says returned. Both records exist, but the system can't yet establish which state applies. Those cases can all produce 'I don't know', but they need different next steps: improve retrieval, obtain the missing record, or resolve a contradiction. For an eval, I'd label the required evidence and expected unresolved state, then check whether the answer names the actual gap. A citation to a relevant document wouldn't count as enough on its own. Neither would refusing every difficult question. I'd also test whether adding the missing evidence changes the answer appropriately, and whether adding a conflicting source makes it less certain. Do your evals distinguish these cases, or do they mainly score the final answer as correct, incorrect or refused?
document fetched 2026-10-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x9roy0/an_unanswered_rag_question_can_mean_three_very