← sources

RAG chatbot looks great in demo but falls apart on real user questions. How are you handling retrieval quality?

source: reddit (reddit:artificial)published 2026-10-13 10:16ingested 2026-10-14 02:36

retained text

Working on a RAG based support bot for a client project and hit a wall I did not expect. On our test questions it looks great, but real users type messy stuff like half sentences, typos, and questions that need info from 2 or 3 different docs. Retrieval keeps pulling the wrong chunks and the model then answers confidently anyway.

What we have tried so far:

* Smaller chunks with overlap (helped a bit, but hurt on long policy docs)
* Hybrid search (BM25 plus embeddings), which was the biggest improvement so far
* A reranker on the top 20 results (better quality, but adds latency)

Still struggling with multi-doc questions, and with getting the bot to say "I don't know" instead of guessing. Has anyone found a reliable way to evaluate this without manually reading hundreds of outputs?

document fetched 2026-10-14 00:20 · canonical https://reddit.com/r/artificial/comments/1x4suc4/rag_chatbot_looks_great_in_demo_but_falls_apart