Agent Evaluation Metric for multi-turn conversations

extract

Artificial Intelligence Agent Evaluation Metric for multi-turn conversations Multi-turn agents fail in ways that single-turn evaluation misses: one early mistake quietly corrupts every later turn. This post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level way to measure agent quality. We apply it to its first dimension, correctness. We show how AEM pinpoints the one turn that caused a failure and separates it from the turns that merely inherited the problem. The correctness challenge in multi-turn agentic conversations Evaluating agent correctness in a multi-turn conversation is hard because the property itself is fragile, and holistic scores hide where it breaks. This section shows why cascading errors defeat outcome-level evaluation and motivates a decomposable, turn-level metric. Why correctness matters A single wrong tool call cascades into downstream failures across turns. Consider a five-turn conversation with an enterprise assistant. The user asks to create a sales report, then refine it. In turn 2, the agent selects the right action but passes “profit” instead of “revenue.” That single error then silently propagates through every subsequent turn. The following diagram traces that failure. It shows one early error in turn 2 cascading through turns 3–5, and how turn-level evaluation isolates the single root cause from its downstream effects. A task-level evaluation checks only the final outcome. It marks the entire interaction as failed without revealing that only one turn needs fixing. This is the core problem: in multi-turn agentic conversations, errors cascade, and outcome-level evaluation can’t tell root causes from their downstream effects. Limitations of existing metrics Most agent evaluation tools score quality holistically, at the task or response level. The current generation provides goal-completion scoring and large language model (LLM)-as-judge quality assessment. Some also add trace-level root cause analysis. These are valuable, but they share a common framing: agent quality is treated as a single signal, not broken down into parts that can be tracked independently. What they don’t provide is a way to decompose quality into named, separately measurable sub-metrics tracked turn by turn. Knowing an agent “scored 70 percent on goal completion” doesn’t tell you whether the failures were factual errors, missing information, or wrong tool choices. A single score also can’t extend cleanly to new dimensions as requirements grow. Three gaps follow: - Task-level metrics (goal success rate) tell you whether the agent completed the task but not which quality dimension broke down. - Single-turn metrics (helpfulness, faithfulness) evaluate responses in isolation without considering how errors propagate across turns. - Holistic scores cannot distinguish a factual error from a missing required field, and they offer no clear path to add new dimensions (safety, instruction retention) without re-architecting the evaluation. A decomposable, turn-level evaluation pattern closes these gaps. It breaks quality into named sub-metrics, evaluates each turn within the full trajectory, and composes them into a single indicator. The same approach extends to new dimensions without changing the mechanism. Correctness as a composite metric We define agent quality as a single composite indicator built from named, separately measurable sub-metrics, computed by AEM. In this first post, AEM measures correctness through two sub-metrics: - Truthfulness: Are the values produced by the agent factually consistent with what was expected? This applies to both parameter values in tool calls and statements in natural language responses. - Completeness: Are all required elements present? No missing parameters, and no partial responses that omit requested information. The following diagram shows this decomposition. It illustrates how a top-level score breaks into named sub-metrics that are measured independently and then recombined, and how the same pattern extends to future dimensions. The headline contribution is the decomposition itself. AEM isn’t a single opaque score but a composite of sub-metrics, each measurable per turn and composable across the trajectory. Tool and action selection form the structural foundation beneath them. The same decompose-evaluate-compose pattern extends to new dimensions such as safety, instruction retention, and reasoning depth. Correctness is the first dimension we instantiate. The Agent Evaluation Metric framework AEM turns the decomposition idea into a concrete, per-turn metric. This section defines the turn-level hierarchy, the two sub-metrics and how they compose, and the failure taxonomy that makes a score actionable. The turn-level hierarchy Correctness is computed per turn. A turn is either a response turn, where the agent replies to the user, or an action turn, where the agent invokes a tool. We lead with the response turn, since that is what the user ultimately sees, and the same metric applies to action turns too. Both fall under one hierarchy. The following diagram shows that shared hierarchy. It shows the same two sub-metrics evaluating a tool call (parameter keys and values) on one side and a natural-language response (coverage and grounding) on the other. For a response turn, the two sub-metrics apply to free text. Completeness asks whether the reply covers the full query, and truthfulness asks whether it’s factually consistent. The same composite applies to an action turn. There, completeness checks the parameter keys, confirming that all required parameters are present. Truthfulness checks the parameter values, confirming they are semantically correct. Both sit on top of a structural check that the right tool and action were selected. For this post, a turn’s correctness is treated as binary: pass or fail, with a specific failure reason naming the sub-metric and the field. The same decomposition also supports finer-grained grading, scoring individual claims or fields on a continuous scale. Composed across a dialog, the AEM score is the proportion of turns that pass. Because it’s decomposed by sub-metric, a decline shows which dimension, truthfulness or completeness, drove the change, not just that correctness dropped. Decomposing and formalizing AEM Both sub-metrics rely on semantic comparison rather than exact matching. For responses and parameter values, exact string matching is too brittle. “New York City” and “NYC” are semantically equivalent, and “Q3 2024 revenue” and “third quarter revenue figures for 2024” convey the same information. Semantic similarity scoring determines whether two values are semantically equivalent. Conceptually, the check looks like this: The scorer can be an embedding-based similarity check (fast, cheap) or an LLM-as-judge call (more nuanced). The embedding check is a transparent encoder-plus-similarity-function, and the judge is a more opaque decoder-based model whose scoring isn’t directly inspectable. The threshold controls strictness: a higher threshold catches real errors but risks flagging semantic equivalents, while a lower one is more permissive. The 0.5 here is a neutral default to start from, not a tuned value. The right value depends on your domain’s tolerance for false positives versus false negatives (see Lessons learned). The similarity score is continuous. The threshold is what collapses it to a binary turn verdict for this post, and a finer-grained setup can retain the per-claim scores instead. Completeness is checked semantically for response turns (does the response address the full question?) and structurally for action turns (are all required parameter keys present?): The returned missing and extra sets feed directly into the failure taxonomy. A non-empty missing set produces a missing_parameters failure, and a non-empty extra set produces an extra_parameters failure, pinpointing exactly which parameters were wrong. Composing the score. Decomposition produces per-sub-metric, per-turn verdicts. Composition turns them into one number. The composition rule is itself a choice, in keeping with the framework’s composable principle. The default in this post is an unweighted mean of passing turns. Other rules are equally valid. Weighted means give more weight to turns that carry higher cost when wrong. Gating rules let a single critical-turn failure cap the score. Per-sub-metric thresholds set a separate bar for each dimension. The framework treats this composition function as pluggable. Failure taxonomy and action chains When a turn fails, a specific failure reason captures exactly what went wrong. The taxonomy spans both turn types: response-turn failures sit at the same level as action-turn failures, and the structural checks (tool and action selection) apply only where a turn invokes a tool. The two sub-metrics, truthfulness and completeness, are the constant across both turn types. Only the structural checks are tool-specific. The prior_action_failed label is what makes the metric actionable in a multi-turn setting. It separates root causes from cascading effects, and it can attach to a response turn as readily as to an action turn. The evaluator assigns the label by dependency. A turn is a root cause when its failure originates in that turn. It gets prior_action_failed when it fails only because it consumed an already-failed turn’s output. In the opening example, only turn 2 is a root cause, and turns 3–5 inherit the label. The metric also tracks action chain length (single call, two-step, and complex three-step-plus sequences), since longer chains concentrate most degradation. Evaluation pipeline for agentic workflows The metric described earlier runs inside a repeatable pipeline. Annotated dialogs go in, and a single decomposed AEM score comes out, attributed per turn and consumed across the agent lifecycle. The following diagram shows that end-to-end flow, from annotated dialogs through per-turn scoring and attribution to a single score consumed in development and production. For the five-turn sales-report example, the pipeline returns a compact, decomposed result (illustrative): Golden dataset design Evaluation begins with ground truth: conversations where both the correct responses and the correct tool calls are annotated. This gold reference is typically human-annotated (or human-reviewed when bootstrapped from a stronger model), since it defines what correct means for each turn. Each dialog is a sequence of turns, and each turn pairs the gold (expected) output with the predicted (actual) output. A turn carries a turn role identifying who produced it. A response turn and an action turn look like this: Comparing gold and predicted yields the per-turn correctness verdict. The response turn passes: the wording differs from gold but is semantically equivalent, which is exactly what semantic similarity scoring is for. The action turn fails: a truthfulness error surfaces on the metric value (profit where revenue was expected). The tags field supports order-invariant evaluation. When multiple tool calls are valid in any order, such as checking a calendar and searching flights, the evaluator checks against valid orderings. It does not penalize correct but differently sequenced behavior. Error attribution After every turn is evaluated, error attribution separates root causes from cascading effects. It operates on turn verdicts regardless of whether a turn was a response or an action: This changes how teams prioritize fixes. Instead of investigating every failure independently, they focus on root causes, since cascading failures often resolve after the root is fixed. A single root cause in a multi-step chain can otherwise appear as several distinct failures. Production monitoring In production, the overall correctness score is tracked continuously: - Overall correctness by release (regression detection): Did the latest model update degrade turn-level success rate? - Correctness by chain length: Do complex multi-step chains degrade over time? - Failure reason distribution (root cause trends): Is tool_mismatch increasing after a model swap? - Latency correlation: Do conversations with higher accumulated latency show lower correctness? The framework outputs structured JSON that feeds into monitoring dashboards. Some errors are costly, such as a financial calculation or a compliance-related response. For those, the decomposed sub-metric scores can also be correlated against human or gold labels. A per-sub-metric correlation (for example, Pearson or Spearman) shows whether the automated score tracks human judgment and where to bring a reviewer into the loop. Integrating with the Strands Agents evaluation SDK The methodology is framework-agnostic, but many teams run evaluations through an existing harness. The turn-level correctness signal integrates with the Strands Agents evaluation SDK as a custom evaluator. From there it plugs into the same pipeline teams already use for goal-completion and LLM-as-judge scoring. Those built-in evaluators report quality as a holistic, per-trajectory signal. AEM is complementary, contributing a decomposed, per-turn correctness score that attributes a failure to a specific sub-metric and turn. The wrapper reuses the per-turn checks built earlier in this post (evaluate_truthfulness, evaluate_completeness, and the attribution logic): Here evaluate_dialog applies the same per-turn truthfulness and completeness checks shown earlier and returns a result per turn. This pattern keeps the evaluation logic (the decomposition, the failure taxonomy, the turn-level composition) as portable custom code, while Strands Agents provides the runner, trace collection, and reporting. Concretely, each run yields per-turn traces (spans for tool calls and model invocations) and a structured report. You can display or export that report as JSON to your own dashboards and alerting. AEM adds the per-turn correctness score. The wrapper runs the correctness signal alongside other evaluators in a single run, including the safety evaluator we introduce in the next post. Applying the framework to Amazon Quick Suite Amazon Quick Suite is an enterprise assistant that runs the kind of multi-turn, multi-tool conversations this metric targets. Rather than report internal production numbers, this section walks through how a team reads AEM output, using the sales-report conversation from the opening example. Reading an AEM score What the user ultimately judges is the response they receive at each turn, so that response is the unit we evaluate, turn by turn across the dialog. Behind a single response, the agent often chains several tool calls under interactive latency constraints, and AEM scores the correctness that results. Running the evaluation pipeline across a dialog produces the single AEM score for the conversation and its decomposition. The success_rate is the leading indicator, alongside the per-sub-metric breakdown and the test_pass flag, which is true only when every turn passes. That same call returns the per-turn failure reasons that drive error attribution. A worked attribution example Return to the five-turn sales-report conversation. Turn 2 passes profit where revenue was expected, so it fails on truthfulness with inconsistent_parameter_values. Turns 3–5 build on that result and fail too, but as cascades: each carries prior_action_failed. A raw failure count reports four broken turns. Attribution reports one root cause at turn 2 and three downstream effects, which is the number that matters. The practical rule is to attribute first, then investigate. The following recurring patterns make this concrete, and the table summarizes where to look in each case. The takeaway is that longer chains push a larger share of failures back to earlier turns rather than to independent errors. Fixing a small number of root causes can resolve many of the observed failures, so attribution turns a noisy failure list into a short, ordered fix list. Lessons learned A few lessons emerged from applying AEM in practice. First, attribute before you investigate: the prior_action_failed label separates root causes from cascades, so one early error in a chain does not read as many independent failures. Second, tune the similarity threshold to your domain. Set it by your tolerance for false positives versus false negatives. Too strict a threshold flags semantic equivalents such as “NYC” versus “New York City,” while too lenient a threshold misses real errors. Third, tag order-invariant steps. When several tool calls are valid in any order, marking them lets the evaluator credit valid alternate orderings instead of failing correct behavior. Two broader lessons concern trust and growth. For costly errors, validate against human labels by correlating the decomposed sub-metrics with human or gold judgments. Bring a reviewer into the loop where an undetected error carries real consequences. And extend with new sub-metrics rather than new pipelines: define per-turn criteria and a failure taxonomy, then compose with the same score. Planning quality, instruction retention, and safety all follow this approach. Conclusion and next steps This post presented the Agent Evaluation Metric (AEM) as a single composite indicator for multi-turn agentic conversations, applied to its first dimension, correctness. AEM decomposes correctness into named sub-metrics (truthfulness and completeness) and evaluates them at the turn level with precise error attribution. It goes beyond detecting failures: it identifies which dimension broke down, which turn caused it, and whether later failures are root causes or cascading effects. The methodology fits into an evaluation workflow a team already runs. Ground truth comes first, then AEM scores each turn through truthfulness and completeness. Error attribution then separates root causes from cascading failures, so the count reflects independent problems rather than downstream noise. Tracking the score across model versions turns the metric into a regression signal, run inside an existing Strands Agents pipeline through the custom-evaluator wrapper. A decline points to the sub-metric responsible, and the failure taxonomy identifies the precise turn to investigate. Correctness is the first dimension AEM instantiates through a deliberately extensible approach: decompose a quality concept into named sub-metrics, evaluate each turn within the full trajectory, attribute failures, and compose the results into a single indicator. The next post in this series applies the same method to safety, and later posts extend it to multilingual and multimodal evaluation. To get started, explore the working examples in the Strands Agents samples repository and the Strands Agents evaluation documentation, then adapt the custom evaluator shown earlier to your own dialogs. To learn more about the enterprise assistant used here, see Amazon Quick Suite. Teams adopting correctness today can add safety and other dimensions as their requirements grow.