Testing AI agent accuracy

retained text

AI/ML
Testing AI agent accuracy
An AI agent context graph management supplier called DataHub asks this question; how do AI agents monitor, prove, and compound their own accuracy to earn sustained trust?
There are underlying questions here, such as: what do we mean by AI agent accuracy? And: how do we test it?
We understand that Holland’s Optiver, a global automated trading firm, saw agent accuracy go from 42 percent to 74 percent on real researcher questions across its petabyte-scale data ecosystem after layering in a governed context facility. Is 74 percent accuracy good enough for production use? Most probably not.
Online collaboration platform Miro has more than 20,000 datasets spread across Snowflake, dbt, and Looker, spanning domains from finance to marketing to product analytics. When Miro connected Claude Code directly to Snowflake to enable natural language querying for business users, the early results were promising for simple questions but note for more complicated ones.
A question like “give me the active users” would return something plausible but wrong. Ronald Angel, Lead Product Manager at Miro, and his team measured this precisely. Using an evaluation bank of 900 questions, testing text-to-SQL accuracy, created manually by data experts across business domains, they benchmarked accuracy at less than 40 percent. A sample question was: “What percentage of our premium accounts have two-factor authentication enabled?”
- Accuracy was measured end-to-end: whether the agent produced the correct answer (right tables, joins, filters, and results) for each question.
- The benchmark ran regularly and was updated with model or pipeline changes.
- When Claude Code was connected directly to Snowflake (with ~20,000+ datasets and no strong context layer), accuracy was under 40 percent . Wrong-table routing occurred in more than 30 percent of cases.
Ronald Angel, Product Manager, Data Platform, Miro, said: “Starting with Snowflake metadata alone, our analytics agent answered about half of our benchmark questions correctly. After layering in DataHub Cloud as our context platform, including data product documentation, cross-source context and business meaning derived from our query history, we nearly doubled accuracy from close to 50 percent to around 90 percent .”
Rather than continuing to connect Claude Code directly to Snowflake with no context layer, Miro placed DataHub between the orchestrator and the SQL engine, using the DataHub MCP server to expose data entity descriptions, tags, glossary terms, column schemas, lineage, and search to the agent.
Miro used Claude Code to orchestrate question decomposition and DataHub MCP to surface governed metadata, with Snowflake Cortex executing the final SQL.
It tracked accuracy as they added DataHub capabilities as a context/routing layer (via DataHub MCP exposing descriptions, tags, glossary terms, lineage, schemas, etc.) in stages:
Reviewing the benchmark questions still failing at the 85 percent level, Miro’s team found that many failures traced not to routing logic but to sparse model documentation or deprecated columns the agent could not distinguish from live ones.
The fix was to reduce the number, of data assets used by the agent and clarify definitions. Specifically, Miro curated only the data products critical to its AI use cases, documenting them with business definitions, dimensions, and measures in dbt, loaded into DataHub via MCP. The remaining 20,000+ assets were intentionally left out of scope.
It said that, when the right asset is well-documented and exposed as the only semantic view the agent can reach, routing errors disappear. The agent does not need to choose between 50 tables. It sees one. This is called a deterministic mode and has >90 percent accuracy.
Read the DataHub Miro case study here.
Takeaways
Before you can assess AI agent accuracy, you have to define it. That’s a simplistic statement but the actuality is complex. Look at Miro with 900 text-to-SQL test questions, and a 4-stage project progressively improving accuracy, with the final stage dispensing with >20,000 non-essential data sets and setting data definitions, etc. in stone as it were. Many team members were involved in this over time. It was a far from a simple task and only applies to one scenario; text-to-SQL generation. And it still didn’t get to 100 percent.
Consider the idea of an AI agent acting on your behalf and booking airline tickets. Would a 90 percent accuracy rate be acceptable? It books a passenger identity, an airline, departure airport, a destination, a departure date, a departure time, a flight class, a seat, identifies a travel rewards program, and pays a price. Which of these ten things could it get wrong and still be acceptable?
None of them, of course. These are all mission-critical, so to speak. We need 100 percent accuracy.
Verifying AI agent accuracy is going to become profoundly important.