← sources

How are you testing AI agents before deploying them to real users?

source: reddit (reddit:AI_Agents)published 2026-09-28 12:53ingested 2026-09-29 02:36

retained text

I am researching how teams evaluate customer-facing AI agents in production.

Traditional LLM evaluation mostly asks:

**"Did the model generate a good response?"**

But production AI agents create bigger questions:

* Did the agent access the correct data?
* Did it call the right tool/API?
* Did it follow authentication and permission rules?
* Did it take the correct action?
* Did it know when to escalate to a human?

Example:

A lending AI agent tells a customer:

"Your outstanding amount is ₹18,400."

The response sounds correct, but how do we verify:

* Was ₹18,400 the actual backend value?
* Was the correct customer authenticated?
* Did the agent retrieve the right account?
* Did voice recognition misunderstand the amount?

For people building AI agents:

**How do you currently test these scenarios before production?**

Do you rely on:

* Manual QA?
* Custom evaluation datasets?
* LLM-as-a-judge?
* Unit tests for tools/functions?
* Observability platforms?
* Production monitoring?

What are the biggest failures you have seen with AI agents after deployment?

document fetched 2026-09-29 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1wsdskk/how_are_you_testing_ai_agents_before_deploying