RAG chatbot looks great in demo but falls apart on real user questions. How are you handling retrieval quality?
retained text
Working on a RAG based support bot for a client project and hit a wall I did not expect. On our test questions it looks great, but real users type messy stuff like half sentences, typos, and questions that need info from 2 or 3 different docs. Retrieval keeps pulling the wrong chunks and the model then answers confidently anyway. What we have tried so far: * Smaller chunks with overlap (helped a bit, but hurt on long policy docs) * Hybrid search (BM25 plus embeddings), which was the biggest improvement so far * A reranker on the top 20 results (better quality, but adds latency) Still struggling with multi-doc questions, and with getting the bot to say "I don't know" instead of guessing. Has anyone found a reliable way to evaluate this without manually reading hundreds of outputs?
document fetched 2026-10-14 00:20 · canonical https://reddit.com/r/artificial/comments/1x4suc4/rag_chatbot_looks_great_in_demo_but_falls_apart