Cost vs. Quality Tradeoff Framework for Agent Models

extract

Cost vs. Quality Tradeoff Framework for Agent Models OpenRouter · Picking a model for an agent by leaderboard rank pays frontier prices for tasks that a cheaper model may handle at the same accuracy. The question to answer is not which model scores highest, but which model is the cheapest one that is good enough for the task in front of you. This guide is a three-step framework for making that call. You set the quality bar the task needs, measure cost per quality point on your own examples, and pick the cheapest model that clears the bar with margin. Tl;dr - Set a quality bar per task before you compare models. A model below the bar is disqualified no matter how cheap it is. - Run a cheap, a mid-tier, and a frontier model on 20 to 50 of your own examples, score them with one rubric, and divide cost by score to get cost per quality point. - Pick the cheapest model that clears the bar by more than the score swing you observe between runs. - Read cost from the usage.cost field in each response rather than multiplying listed rates by estimated token counts. - Rerun the comparison when a candidate model or its price changes. What a leaderboard rank does not tell you A leaderboard averages a model’s results across tasks that have nothing to do with yours. The model that ranks first for coding is not necessarily first at your structured-data extraction task, and a mid-tier model that looks unremarkable on a reasoning benchmark may answer your FAQ traffic accurately enough for your bar. Agent tasks are often narrow, such as classifying a ticket, extracting one field, or escalating when the model is unsure. Whether a cheaper model reaches the same accuracy as a frontier model on a narrow task is a measurement, and the leaderboard does not make it for you. Cost for an agent is also more than one prompt and one response. A single chat completion is billed once. An agent pays for every tool call, every intermediate step, and every retry. A three-step loop pays the per-token price at least three times before it returns an answer. If you pick by leaderboard rank, you can pay frontier prices three times over for a task that a cheaper model would have handled at the same accuracy. Step 1: Define the quality bar the task needs Before you compare anything, decide what good enough means for this task. The bar is different for every job, and it is the filter every later step runs through. If a wrong answer is a liability, as in compliance, healthcare, or legal review, set the bar high and accept the higher cost per request. If the task is high-volume support or chat, the aggregate outcome matters more than any single response. A cheaper model that resolves 90% of routine requests correctly and escalates the remaining 10% cleanly may be acceptable for that task. You decide where the bar sits, and the framework measures against it. Latency-sensitive work, such as fraud checks and live chat, adds a third constraint. A model that is cheap and accurate but too slow for the task is disqualified before cost or accuracy come up. Speed, cost, and accuracy can’t all be maximized at once. Decide which constraints your task has, and the candidate list narrows. If you don’t know where a task sits, one starting point is to run a mid-tier model for everything, then split tasks by where the measurements show a problem. Tasks where the mid-tier model is more than you need move to something cheaper. Tasks where it misses the bar move up. Measurement decides which tier each task needs instead of a guess made up front. Step 2: Measure cost per quality point on your own examples Pull the documents, tickets, and prompts your agent will really see. Use your own traffic, not a public dataset and not benchmark examples. The point is to measure your task. Run a cheap option, a mid-tier option, and a frontier option across those examples. Score them with one consistent rubric. For deterministic tasks like classification, use exact match against a fixture, which fits when exactly one value is correct. For open-ended tasks, use an LLM as a judge to score quality. Our LLM-as-a-judge guide walks through setting that up. Scoring is easier when every candidate returns the same shape of output, which is what structured outputs and response_format are for. Divide the model’s cost per 1,000 requests by its quality score and you have a cost per quality point you can compare directly across candidates and across test sets of different sizes. Run this as a short script that swaps one model string and reads the per-request cost from the usage object in each response. We return the usage object in every non-streaming response and in the final message of every streaming response without any extra request parameter, as described in the usage accounting docs. usage.cost is the amount we charged your account for that request, in USD. The OpenAI Python SDK keeps response fields it doesn’t define, so r.usage.cost is available as an attribute. import json import os from openai import OpenAI client = OpenAI( base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"], ) ROUTING_PROMPT = open("routing_prompt.txt").read() test_set = json.load(open("test_set.json")) # 20-50 of your own {"ticket": ..., "label": ...} examples candidates = [ "openai/gpt-5.6-luna", "google/gemini-3.7-flash", "anthropic/claude-opus-5", ] for model in candidates: total_cost, correct = 0.0, 0 for ex in test_set: r = client.chat.completions.create( model=model, messages=[ {"role": "system", "content": ROUTING_PROMPT}, {"role": "user", "content": ex["ticket"]}, ], ) total_cost += r.usage.cost answer = (r.choices[0].message.content or "").strip() correct += int(answer == ex["label"]) score = correct / len(test_set) points = score * 100 cost_per_1k = total_cost / len(test_set) * 1000 cost_per_point = cost_per_1k / points if points else float("inf") print(f"{model}: score={score:.0%} cost/1K=${cost_per_1k:.2f} cost/point=${cost_per_point:.4f}") A response can carry no text content, for example when the model refuses, so the script treats missing content as an empty answer, counts it as incorrect, and keeps its cost in the total. A model that scores zero has no cost per point, so the script prints inf for it instead of dividing by zero. Here is a worked example for a support-routing task. Assume each request sends about 700 input tokens, the ticket plus a short system prompt, and returns about 150 output tokens. Cost per 1,000 requests is (700 / 1M × input price) + (150 / 1M × output price), multiplied by 1,000. Cost per quality point is that cost divided by the quality score. The prices are the listed rates in our model catalog on 2026-09-18 for GPT-5.6 Luna, Gemini 3.7 Flash, and Claude Opus 5. They are the model-level prices the catalog lists. Individual provider endpoints and the flex and priority service tiers bill at their own rates, and GPT-5.6 Luna bills a higher rate on requests with 272,000 or more prompt tokens. The token counts and quality scores are illustrative. Your own run replaces them. Read the table in the order the framework runs. First gate on the bar. At 85%, GPT-5.6 Luna is out at 82, so its low cost per quality point doesn’t matter. A model below the bar is disqualified no matter how cheap it is per point. That leaves Gemini 3.7 Flash and Claude Opus 5. Between the two that clear, take the cheaper one. Gemini 3.7 Flash costs $1.09 per 1,000 requests. Claude Opus 5 costs $7.25 for a score the task didn’t require. Gemini 3.7 Flash is the pick, and it clears with a 4-point margin. Change the situation and the answer changes. If a misroute means a missed SLA and you set the bar at 95%, only Claude Opus 5 clears and you pay the $7.25. The goal is not to pick the highest score. It is to know which model clears your bar for the least money, and to see the gap you pay for when you reach past it. Step 3: Pick the cheapest model that clears the bar with margin Pick the cheapest model that clears the bar with margin, not the model that wins outright. The margin matters because the numbers move. Model scores drift as providers update weights, your own traffic shifts over time, and new versions change the picture. Set the margin from observation rather than picking a fixed percentage. Run the candidates more than once, or across a fresh slice of traffic, record how much the score moves between runs, and require the winner to clear the bar by more than that observed swing. A model that only reaches the bar on a single small sample should have to clear a higher internal target before you commit to it, so that ordinary variation does not push it below the bar in production. Applied across a few tasks, the framework looks like this. The cost figures use the same listed prices as the table above at the input and output token sizes noted per row. The quality bars are examples, and the cost you record comes from usage.cost. The method is the same in each row, and the answer differs because the bar differs. The measured tradeoff was true on the day you measured it. Recheck it when a newer model is released, because the model that cleared your bar yesterday may now cost the same as one that sits just below it. Recheck the comparison when models or prices change New models are released often, and prices on existing models change. Between May and September 2026, four versions of Gemini Flash were added to our catalog: Gemini 3.5 Flash on 2026-05-19, Gemini 3.6 Flash on 2026-07-21, Gemini 3.7 Flash on 2026-08-13, and Gemini 3.8 Flash on 2026-09-02. A comparison you ran a few months ago can already be out of date, so rerun it whenever a candidate is updated or its pricing changes. Rerunning is cheap on our side. We expose one OpenAI-compatible API, so swapping a model is a config change. The base URL, key, and SDK stay the same. You change the model string from openai/gpt-5.6-luna to anthropic/claude-opus-5 and the script above runs against the new model. You don’t write a new integration for each provider you want to test, which is what makes it practical to rerun your examples against a new release the week it comes out. That same one-line swap is what model routing in a multi-agent setup relies on. Two things keep the framework grounded in real numbers. We show per-model pricing in the model catalog before you commit, so the cost side of your comparison starts from live data. And because every response reports usage.cost, the spend you measured is what we billed, not an estimate from a rate card. Frequently asked questions What is the cost versus quality tradeoff for AI models? It is the tradeoff between what a request costs and how well a model performs on your task. Speed is a third constraint. A model that is cheap and accurate but too slow for the task is still out. Choosing a model means deciding how much quality the task needs and paying for that, rather than paying for the highest score available. What is the best model for AI agents? There is no single best model. The right model for an agent depends on the quality bar its task has to clear, the latency it has to respond within, and the volume it has to handle. A model that clears the bar for compliance review costs more than a task like support triage needs, and a model that is cheap enough for triage may not clear the bar for compliance review. How are AI models priced for agent workloads? Text models in our catalog list a prompt rate and a completion rate per token. Some endpoints also list rates for cached input, reasoning tokens, per-request charges, and flex or priority service tiers. Agent workloads add tool calls and retries on top of the first request, so the token price of one completion understates the cost of one task. Measure cost across a full agent run and read it from usage.cost in each response. Is a multi-agent design more or less expensive than a single agent? It depends on how the work splits. A multi-agent design can send routing and classification to a cheap model and call a more expensive model only for the requests that need it, which costs less than sending every request to the expensive model. It also adds requests, so measure the full run rather than assuming the split saves money. Conclusion Define the bar, measure complete runs on your own examples, and choose the lowest-cost model that clears the bar by more than the swing you observed between runs. Keep the script and the example set so you can rerun the comparison when models, prices, or your traffic change. References - Usage accounting for the usage.cost field returned in every response. - Structured outputs for response_format and JSON schema responses. - Service tiers for flex and priority pricing. - Model catalog for current per-model prices. - LLM-as-a-Judge: Score AI Agent Outputs Automatically for scoring open-ended tasks. - How OpenRouter Model Routing Works for routing between models.