# How do teams catch AI chatbots giving customers wrong answers? — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/ea91ba2c14123a33

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-11-19 14:22 ingested 2026-11-20 02:35

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1y0akr8/how_do_teams_catch_ai_chatbots_giving_customers/)

## retained text



```
Our gym's chatbot told a member she could pause her membership for three months for free, which we've never offered, and she posted the screenshot in the town's facebook  group

The bot now runs on Orq Ai and since then we've set up an evaluator that checks every answer against our membership terms and runs as a guardrail, so a failing answer goes to staff instead of reaching the member  
we tested it on old transcripts in an experiment first

The problem is that it's either too strict or too loose   
strict cause it blocks normal questions like bringing a friend on saturday cause the wording doesn't match the terms doc loose  causeit let the bot promise a free locker

I'm thinking of only checking answers that say yes to something, since every invented policy so far has started with yes. If your bot catches made-up promises without handing off every boring question, what are you checking?
```



document fetched 2026-11-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1y0akr8/how_do_teams_catch_ai_chatbots_giving_customers
