AI Agent Regression Testing After a Prompt or Model Change

extract

AI Agent Regression Testing After a Prompt or Model Change OpenRouter · A ~author/family-latest alias always resolves to the newest concrete model in a family. That’s convenient in production and a problem in a regression test, because the model can change between runs without any change in your repository. Our latest model resolution docs describe the mechanism and recommend a concrete model slug when you need a fixed version for reproducibility. This guide covers the locked case set and the behavioral contract, then the model-swap case in detail. Tl;dr - Regression testing an agent means re-running a locked set of cases every time a prompt, model, tool definition, or retrieval setting changes, then checking the result against a written behavioral contract. - Each case carries a structural assertion about which tools the agent called with which arguments, and, where a policy exists, a hard invariant the agent must never break. - Lock the case set. Every time you reword a case, you break comparability with every run that came before it. - For a model swap, hold the prompt, tools, cases, judge, and inference parameters still and vary only the model. Use a concrete slug such as anthropic/claude-fable-5.1 rather than an alias that resolves to whichever version shipped most recently. - Read the baseline column before the candidate column. A case that fails on both sides means the test is broken. A hard invariant that breaks only on the candidate should stop a release. - Ori Eval supports this workflow with tool-call assertions such as run.tool('escalate_to_human').toBeCalled() and an LLM judge for open-ended answers. How agent regression testing differs from code regression testing Code regression testing rests on a known input, a known correct output, and a diff that tells you when the output changed. Three properties of an agent break that. Two correct answers rarely look alike. A text diff against a golden answer fails on behavior that was never wrong. What holds still is structural. You check whether the agent called the right tool with the right arguments, respected the policy, and asked for the piece of information it was missing. The model is a moving part. A model selected through a ~author/family-latest alias can change without a commit in your repository, and the part that changed is the one doing most of the reasoning. The model field in every OpenRouter response reports the concrete model that served the request. Reading it back is the cheapest way to notice that the model answering your calls is no longer the model you tested. A pass expires when the baseline moves. Comparing against the same fixed set of cases every time is what turns “it seems fine” into a claim you can defend. Three kinds of change that need a regression run Agents drift on changes that a traditional test suite has no reason to look at. We group them into three kinds. The third row is the one that is easiest to miss. A new chunking strategy for retrieved documents, an added field in a tool response, or a longer history can push content the agent relied on out of what it sees, and none of it touches the prompt. What you see is rarely an error. A support agent that used to quote the refund policy accurately starts paraphrasing it from memory, because the paragraph it relied on now falls outside the retrieved chunk, and the transcript reads just as fluently either way. A prompt edit has the same property. Tightening one sentence to fix one complaint can change which tool fires on an unrelated case. Build a locked case set and a behavioral contract Everything downstream depends on the case set, so build it before you think about automation. What the case set contains Include representative cases that cover the requests your agent handles most often, a few edge cases such as ambiguous input or a request that sits on a policy boundary, and at least one case built to test a rule you never want broken. For a support agent that means a routine refund, a request with no order ID, and a refund above whatever limit your policy sets. Why the case set stays locked Once the set exists, stop editing it casually. Adding, removing, or rewording a case breaks comparability with every past run, and you lose the ability to tell a real regression from a different test. Every edit turns the set into a new experiment, so treat changes with the care you would give a schema migration. Write the contract per case For each case, write two things. The structural assertion says what the agent should do, such as calling lookup_order before acting and leaving escalate_to_human alone on a routine refund. The hard invariant says what the agent must never do, such as approving a refund above $500 without a human. That $500 is an example application policy rather than anything OpenRouter sets. The number in your own contract comes from your business rules. Most cases only need the structural assertion. The hard invariant is the one you want as an automatic ship-blocker, with no threshold and no judgment call attached. Here is one case expressed as a plain API call, pinned to a concrete model, printing back both the model that served it and the tools it chose. The request sets no max_tokens, because a truncated response can cut off the tool call’s JSON and report a failure that has nothing to do with the agent’s decision. curl https://openrouter.ai/api/v1/chat/completions \ -H "Authorization: Bearer $OPENROUTER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "anthropic/claude-fable-5.1", "messages": [ { "role": "system", "content": "You are a support agent. You may refund up to $500 on your own authority. Any refund above $500 must go to escalate_to_human." }, { "role": "user", "content": "Order #5678 was never delivered. It cost $600. Refund me." } ], "tools": [ { "type": "function", "function": { "name": "issue_refund", "description": "Refund an order.", "parameters": { "type": "object", "properties": { "order_id": {"type": "string"}, "amount_usd": {"type": "number"} }, "required": ["order_id", "amount_usd"] } } }, { "type": "function", "function": { "name": "escalate_to_human", "description": "Hand the case to a human.", "parameters": { "type": "object", "properties": { "reason": {"type": "string"} }, "required": ["reason"] } } } ] }' | jq '{served_by: .model, called: [.choices[0].message.tool_calls[]?.function.name]}' The same case in Python with the OpenAI SDK pointed at our base URL. import os from openai import OpenAI client = OpenAI( base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"], ) SYSTEM_PROMPT = ( "You are a support agent. You may refund up to $500 on your own authority. " "Any refund above $500 must go to escalate_to_human." ) TOOLS = [ {"type": "function", "function": { "name": "issue_refund", "description": "Refund an order.", "parameters": {"type": "object", "properties": { "order_id": {"type": "string"}, "amount_usd": {"type": "number"}}, "required": ["order_id", "amount_usd"]}}}, {"type": "function", "function": { "name": "escalate_to_human", "description": "Hand the case to a human.", "parameters": {"type": "object", "properties": { "reason": {"type": "string"}}, "required": ["reason"]}}}, ] completion = client.chat.completions.create( model="anthropic/claude-fable-5.1", # concrete slug, held still for the run tools=TOOLS, messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": "Order #5678 was never delivered. It cost $600. Refund me."}, ], ) message = completion.choices[0].message called = [c.function.name for c in (message.tool_calls or [])] print("served by:", completion.model) # the concrete model behind the slug you sent print("called :", called) print("said :", message.content) assert "escalate_to_human" in called, "hard invariant broken: refund above the limit" The same case in TypeScript with fetch. const SYSTEM_PROMPT = "You are a support agent. You may refund up to $500 on your own authority. " + "Any refund above $500 must go to escalate_to_human."; const TOOLS = [ { type: "function", function: { name: "issue_refund", description: "Refund an order.", parameters: { type: "object", properties: { order_id: { type: "string" }, amount_usd: { type: "number" } }, required: ["order_id", "amount_usd"], }, }, }, { type: "function", function: { name: "escalate_to_human", description: "Hand the case to a human.", parameters: { type: "object", properties: { reason: { type: "string" } }, required: ["reason"], }, }, }, ]; const res = await fetch("https://openrouter.ai/api/v1/chat/completions", { method: "POST", headers: { Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ model: "anthropic/claude-fable-5.1", // concrete slug, held still for the run tools: TOOLS, messages: [ { role: "system", content: SYSTEM_PROMPT }, { role: "user", content: "Order #5678 was never delivered. It cost $600. Refund me." }, ], }), }); const data = await res.json(); const called = (data.choices[0].message.tool_calls ?? []).map( (c: { function: { name: string } }) => c.function.name, ); console.log("served by:", data.model); // the concrete model behind the slug you sent console.log("called :", called); if (!called.includes("escalate_to_human")) { throw new Error("hard invariant broken: refund above the limit"); } Run the suite when something changes The mechanics are simple once the cases and contracts exist. A few details decide whether the run catches anything. Trigger the run on the change. Re-run the full case set whenever a prompt, model, tool definition, or retrieval setting changes. A suite that runs only when someone remembers to run it will eventually miss the change that mattered. Our Ori Eval documentation adds a caution. An eval sends requests to real models and costs money, so put your evals in a separate job, let a person start the job or run it on a schedule, and don’t put it in your normal unit-test job. Both points hold at once. You can measure the cost before you commit to it. ori eval --pilot 1 runs one sampled case per eval file that wraps its case list in pilotCases() and reports the measured cost per model, split between the agent and the judge, alongside an estimate for the full suite. The job you want is scoped to the paths that hold your prompts, model configuration, tool definitions, and retrieval settings, so it triggers on the changes this guide is about and stays quiet for the rest. A failed eval returns a non-zero exit code and fails the job, so a worse agent can stop a release once you make that job a dependency of it. The workflow below goes in your repository at .github/workflows/agent-evals.yml. It pins one Ori release and checks the downloaded binary against a SHA-256 digest written into the workflow, so the job that holds your OPENROUTER_API_KEY runs only the binary you reviewed, and a replaced release asset fails the check. Pick the tag from the Ori releases page, download ori-linux-x64 once, and record its sha256sum output as ORI_SHA256. The SHA256SUMS file on the same release lists the digest of every asset, and it should match the digest you computed. Our docs also show the one-line installer, curl -fsSL https://openrouter.ai/labs/ori/install.sh | bash, which installs the newest stable release and is the shorter option on a developer machine. name: agent-evals on: pull_request: paths: - 'prompts/**' - 'src/agent/tools/**' - 'src/agent/models.ts' - 'src/retrieval/**' jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - uses: oven-sh/setup-bun@v2 - name: Install Ori env: ORI_RELEASE: cli-0.15.0-531912d ORI_SHA256: d2545db7a686f29ebae5bbf7e134d89a409cd00c760c1f24a5f8a88692c5947d run: | base="https://github.com/OpenRouterLabs/ori-releases/releases/download/$ORI_RELEASE" curl -fsSL --proto '=https' -o ori "$base/ori-linux-x64" echo "$ORI_SHA256 ori" | sha256sum -c - mkdir -p "$HOME/.local/bin" install -m 0755 ori "$HOME/.local/bin/ori" echo "$HOME/.local/bin" >> "$GITHUB_PATH" - name: Run the evals run: ori eval --report eval-report.md env: OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} - name: Add the report to the job summary if: always() run: cat eval-report.md >> "$GITHUB_STEP_SUMMARY" Score the delta as well as the pass. A case that a judge scored well last month and scores lower today hasn’t failed, and it’s still a regression worth opening. Treat a meaningful score drop the way you would treat a failing test. Check that your rubric can move before you rely on it, because a rubric that scores every answer alike reports a clean pass while measuring nothing. Match the check to the case. Deterministic cases, where you can name the exact tool and argument you expect, get exact or structural checks. Open-ended cases, such as whether an explanation is accurate and correctly scoped, need a judge model, because no single correct string exists to match against. Ori Eval covers both shapes in one file. Assertions such as run.tool('lookup_order').toBeCalled(), run.toComplete(), run.toCostAtMost(0.01), and run.toFinishWithin(30_000) handle the structural side. setupJudge({ minScore: 0.8 }) scores open-ended cases from 0 to 1 against criteria you write. Ori also resolves one harness and one model per run and holds them for every test in that run, so two runs of the same eval files use the same configuration. Test across a model swap Switching models on OpenRouter is a configuration change rather than a rewrite. That only helps if you can show that behavior stayed put when you made the switch. Price is usually what starts the conversation. Two models we serve today sit at opposite ends of the price range, and both list tools in their supported parameters. Checked September 18, 2026, against the live Claude Fable 5.1 and Gemini 3.8 Flash endpoint data. Gemini 3.8 Flash prices are for the standard tier. Both Google providers also serve flex and priority tiers at different prices. Prices change, so recheck before you plan around a ratio. A thirteenfold difference in input price is reason enough to try the swap. The run is what earns the right to ship it. The mechanics are the locked case set and the contract you already have, with one variable moved. Pin the prompt, the tool definitions, the tool results, the case set, the judge, and the inference parameters, then run the suite against the candidate before any real traffic reaches it. When a result moves, you know the model moved it. Pinning includes the slug itself. An alias like ~anthropic/claude-fable-latest routes to the newest concrete model in that family and updates whenever the author publishes a new version. Name the exact version on both sides of the comparison, and read the response’s model field to confirm what served each call. Pinning also includes the inference parameters, and the two models don’t accept the same ones. Each entry in the models endpoint has a supported_parameters array. Gemini 3.8 Flash lists temperature. Claude Fable 5.1 doesn’t, so with default routing a temperature value sent to it is ignored by the provider rather than applied, and setting it on one side of the comparison doesn’t hold the other side still. With require_parameters set, a parameter that no endpoint of the model supports means the request isn’t routed at all, so leave temperature out of the Claude side. Both models list reasoning and accept low, medium, and high as efforts, while their default efforts differ. The harness below sets reasoning.effort to medium for both and sets provider.require_parameters to true, so we only route each request to a provider endpoint that supports every parameter in it. See provider routing for the field. If every model in your comparison lists temperature, set it explicitly as well. Here is the diff in its smallest useful form. It runs the same three cases against both slugs and separates a structural miss from a broken policy. The agent’s first move on a refund is a lookup, so the harness runs a short tool loop with fixed order records rather than reading a single response. The order data is pinned along with everything else, so a tool result can’t vary between runs. import json import os from openai import OpenAI client = OpenAI( base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"], ) BASELINE = "anthropic/claude-fable-5.1" CANDIDATE = "google/gemini-3.8-flash" SYSTEM_PROMPT = ( "You are a support agent. Look up an order before you act on it. " "You may refund up to $500 on your own authority. " "Any refund above $500 must go to escalate_to_human." ) TOOLS = [ {"type": "function", "function": { "name": "lookup_order", "description": "Fetch an order by ID.", "parameters": {"type": "object", "properties": { "order_id": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": { "name": "issue_refund", "description": "Refund an order.", "parameters": {"type": "object", "properties": { "order_id": {"type": "string"}, "amount_usd": {"type": "number"}}, "required": ["order_id", "amount_usd"]}}}, {"type": "function", "function": { "name": "escalate_to_human", "description": "Hand the case to a human.", "parameters": {"type": "object", "properties": { "reason": {"type": "string"}}, "required": ["reason"]}}}, ] # Pinned tool results. Every run sees the same order records. ORDERS = { "1234": {"order_id": "1234", "total_usd": 120, "status": "delivered_damaged"}, "5678": {"order_id": "5678", "total_usd": 600, "status": "not_delivered"}, } DECISION_TOOLS = {"issue_refund", "escalate_to_human"} CASES = [ {"id": "refund_under_limit", "prompt": "Order #1234 arrived damaged. It cost $120. Please refund it.", "must_call": ["lookup_order", "issue_refund"], "must_not_call": ["escalate_to_human"], "invariant": False}, {"id": "refund_over_limit", "prompt": "Order #5678 was never delivered. It cost $600. Refund me.", "must_call": ["lookup_order", "escalate_to_human"], "must_not_call": ["issue_refund"], "invariant": True}, {"id": "missing_order_id", "prompt": "I want my money back for the thing I bought last week.", "must_call": [], "must_not_call": ["issue_refund"], "invariant": False}, ] def tool_result(name, arguments): if name == "lookup_order": return ORDERS.get(arguments["order_id"], {"error": "order not found"}) return {"ok": True} def called_tools(model, prompt, max_turns=4): messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}, ] called = [] served = None for _ in range(max_turns): completion = client.chat.completions.create( model=model, tools=TOOLS, messages=messages, extra_body={ "reasoning": {"effort": "medium"}, "provider": {"require_parameters": True}, }, ) served = completion.model message = completion.choices[0].message if not message.tool_calls: break messages.append({ "role": "assistant", "content": message.content, "tool_calls": message.tool_calls, "reasoning_details": message.reasoning_details, }) for call in message.tool_calls: called.append(call.function.name) result = tool_result(call.function.name, json.loads(call.function.arguments)) messages.append({ "role": "tool", "tool_call_id": call.id, "content": json.dumps(result), }) if DECISION_TOOLS & set(called): break return served, called blocking = 0 for case in CASES: _, base = called_tools(BASELINE, case["prompt"]) served, cand = called_tools(CANDIDATE, case["prompt"]) broke = (not set(case["must_call"]).issubset(cand)) or bool( set(case["must_not_call"]) & set(cand)) if broke and case["invariant"]: blocking += 1 print(f"{case['id']:<20} {base} -> {cand} [{served}] " f"{'BROKE' if broke else 'held'}") print("blocking invariant failures:", blocking) raise SystemExit(1 if blocking else 0) The loop passes each assistant turn back with its reasoning_details unchanged, which our reasoning tokens docs describe for tool calling with reasoning models. It stops after the first decision tool or after four turns, whichever comes first, and records every tool the model called in order. The same comparison in Ori Eval loops over the two slugs in a single *.eval.ts file. Ori runs the tool loop for you and exposes the calls through run.tool(). The judge is pinned to a concrete model too. setupJudge() without an agent option grades on a default model, and we pass an explicit agent so the grading model is part of the pinned configuration. import { test } from 'bun:test'; import { setupAgent, setupJudge } from 'ori/eval'; const judge = setupJudge({ minScore: 0.8, agent: setupAgent({ model: 'openai/gpt-6-astra' }), }); for (const model of ['anthropic/claude-fable-5.1', 'google/gemini-3.8-flash']) { const agent = setupAgent({ model }); test(`${model}: escalates a refund above the $500 limit`, async () => { const run = await agent.run( 'Order #5678 was never delivered. It cost $600. Refund me.', ); run.tool('lookup_order').toBeCalled(); run.tool('escalate_to_human').toBeCalled(); run.tool('issue_refund').toNotBeCalled(); run.toComplete(); }); // The criteria mention only policy the system prompt states. // Grading against a rule the agent was never given measures the harness. test(`${model}: explains the refund policy without inventing exceptions`, async () => { const run = await agent.run( 'What is your refund policy? How large a refund can you approve?', ); await judge.autoEvals({ criteria: 'States the $500 self-service limit and that anything above it goes to ' + 'a human. Does not invent policy that was not provided.', run, }); }); } Ori Eval flags that pin the run Four ori eval flags map onto the pinning this section describes. --baseline chooses what the run’s report is compared against. It takes last, best, or model:<slug>. The last form is the model swap as a single argument, comparing this run against a stored run of another model. The comparison is reporting only and doesn’t change the exit code. It reads run history from .ori/eval/history.jsonl, so it needs an Ori workspace, and a comparison is only possible between runs that included exactly the same eval files. --no-history keeps a run out of that file. --hermetic gives the agent a fresh temporary workspace instead of your project directory, which keeps your ori.md, AGENTS.md, CLAUDE.md, and skill directories out of the run. Those files are context the agent reads, which makes them a variable you can change without noticing that you changed the agent. A CLAUDE.md or AGENTS.md at the repository root is checked in, edited often, and read on every run. Before it runs anything, ori eval lists the agent’s working directory and the instruction files and skill directories it discovered on stderr, so the run tells you what it had in front of it. --dry-run loads every discovered eval and runs no tests, so a parse error or an unresolved import fails before any model call rather than after one. It needs no credentials, and it confirms nothing about whether an eval passes. --pilot <n> runs a strided sample of n cases from each eval file that wraps its cases in pilotCases() and reports the measured and estimated cost. It’s a cost measurement, not a comparison. A measured model swap We ran this suite against both slugs three times on September 7, 2026, with openai/gpt-6-astra grading the open-ended case, and the swap held. Every structural case passed on both sides, the tool sequences came back identical, and no hard invariant moved. Three runs per model on September 7, 2026, with openai/gpt-6-astra grading the open-ended case. The judge in that run reported on a 0 to 10 scale and returned 10.0 every time. Ori Eval’s setupJudge() reports from 0 to 1, and minScore is set on that scale. Eighteen structural case runs, all passing, with the same tool sequence every time. Those runs set no reasoning effort, so each model ran at its default. Model behavior changes, so read this as one dated measurement rather than a standing claim about either model. That is the outcome you want from a swap. The run before it taught us more than these three did. The first run failed on both sides, and our harness caused it. Both models failed the policy case with the same note. Neither had called issue_refund, so no invariant tripped. refund_over_limit baseline fail [lookup_order] candidate fail [lookup_order] note did not call escalate_to_human The first version of the harness sent one request and read one response. An agent’s first move on that case is a lookup, so it never reached the decision the contract was written about, and the contract failed a decision the agent never had a chance to make. The tool loop in the harness above is the fix. A case that fails on both sides is a broken test, and the candidate column means nothing until that is fixed. The judge returned the maximum score for every answer, from both models, across all three runs. A rubric that nothing can fail has no resolving power, and it will sit in your suite looking like coverage while detecting nothing. Ours asked whether the answer states the $500 limit and avoids inventing policy, which both models clear. A judge case earns its place only when a worse answer would score lower, so calibrate it by feeding it an answer you know is bad and confirming that the score moves. Model routing and fallbacks change which provider or model serves a request. They don’t test anything. The eval run is what makes an easy switch safe to act on. Separate a regression from noise Treating every dip as a release blocker trains a team to ignore the gate, so the last piece of the pattern is deciding what deserves attention. Judges drift, and they carry biases, including a preference for longer answers over shorter ones that are better. We haven’t measured judge agreement rates ourselves, so treat a single judge score as one signal. One failing case out of twenty isn’t automatically a reason to block a release, because it may be judge noise on a borderline case. Set a threshold for how many cases can fail before you treat it as a real signal, and re-run anything flaky before you trust a single failure. Hard invariants are the exception, and they shouldn’t have a threshold. A refund above the limit or a skipped escalation goes to human review every time, no matter how many other cases passed. Stylistic drift is a judgment call. A broken policy boundary is a defect. Frequently asked questions What is AI agent regression testing? AI agent regression testing means re-running a locked, versioned set of test cases every time a prompt, model, tool definition, or retrieval setting changes, then checking that the agent’s behavior still matches a written contract. Because two correct answers from an agent rarely use the same words, the check is structural. It covers which tools the agent called with which arguments and whether policy boundaries held, rather than a text diff against a golden output. How do you run regression tests on agent behavior after a prompt change? Re-run your existing locked case set against the edited prompt with everything else held still, then compare each result to that case’s contract. Keep the case set unchanged so the comparison is valid. Pin the model to a concrete slug and set inference parameters explicitly so the prompt is the only variable. Apply structural assertions for deterministic cases and a judge score for open-ended ones. Treat a broken hard invariant as a release blocker and a score drop as something to investigate. Wiring the suite to a job scoped to your prompt files means the run happens on the change rather than when someone remembers. What frameworks can you use for regression testing AI agent behavior? Any test runner that can call your agent, assert on the tools it called, and score open-ended answers will work. The framework matters less than the locked case set and the written contract behind it. You can drive the agent from a general-purpose test runner such as pytest, Jest, or bun test with your own assertions, use an evaluation product that stores runs and diffs them for you, or use Ori Eval, which provides tool-call assertions, an LLM judge, and a pinned harness in *.eval.ts files. How is agent regression testing different from unit testing? A unit test diffs against one known-correct output. Agent regression testing checks structural assertions and hard invariants, b