Do your agent evals include metric definitions changing?
retained text
Suppose a support dashboard changes resolution time from business hours to elapsed hours. The field name stays the same, and both versions say 'hours'. Tickets opened Friday and closed Monday now look much slower, even if the underlying work hasn't changed. An agent comparing the dashboard before and after a release could confidently explain a regression that came from the reporting definition. I'd want the retrieved measurements to carry the calculation definition and version, the relevant calendar and timezone, and when that definition applied. A citation to the dashboard proves where the number came from; it doesn't establish that the two numbers measure the same thing. If the underlying timestamps are available, the agent could request a consistent recalculation. Otherwise it should flag the break in comparability instead of inventing an operational cause. A small eval could hold the ticket timestamps fixed and change only the reporting definition. Then keep the definition fixed and make the tickets take longer. Does the agent distinguish those cases, or produce the same explanation for both? Where do you keep metric definitions so they travel with the data into agent context?
document fetched 2026-10-24 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xda266/do_your_agent_evals_include_metric_definitions