# RAG chatbot looks great in demo but falls apart on real user questions. How are you handling retrieval quality? — retained record

> Retained reddit record from reddit:artificial.

Canonical: https://highsignal.app/data/records/e53641dd2162f391

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:artificial)  published 2026-10-13 10:16 ingested 2026-10-14 02:36

[original source ↗](https://www.reddit.com/r/artificial/comments/1x4suc4/rag_chatbot_looks_great_in_demo_but_falls_apart/)

## retained text



```
Working on a RAG based support bot for a client project and hit a wall I did not expect. On our test questions it looks great, but real users type messy stuff like half sentences, typos, and questions that need info from 2 or 3 different docs. Retrieval keeps pulling the wrong chunks and the model then answers confidently anyway.

What we have tried so far:

* Smaller chunks with overlap (helped a bit, but hurt on long policy docs)
* Hybrid search (BM25 plus embeddings), which was the biggest improvement so far
* A reranker on the top 20 results (better quality, but adds latency)

Still struggling with multi-doc questions, and with getting the bot to say "I don't know" instead of guessing. Has anyone found a reliable way to evaluate this without manually reading hundreds of outputs?
```



document fetched 2026-10-14 00:20 · canonical https://reddit.com/r/artificial/comments/1x4suc4/rag_chatbot_looks_great_in_demo_but_falls_apart
