loading
loading
Artificial Intelligence Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload Organizations building generative AI applications usually compare models the same way: dollars per million tokens. It’s the number on every pricing page, so it becomes the number in every spreadsheet. But production workloads don’t buy tokens. They buy outcomes: a resolved support ticket, a completed research brief, a correct financial summary. Between the pricing page and the outcome sit multipliers the sticker price ignores: how often the model is right, how many tokens it needs to get there, and, for agentic workloads, how many turns it takes, because every turn re-sends the growing conversation. In this post, we share results from an open-source benchmarking harness that measures those multipliers across OpenAI models on Amazon Bedrock (gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol) and two widely used cost-efficient models on the OpenAI API (gpt-5.4-mini and gpt-5.4-nano). We chose the latter two as the cost-optimized baselines many teams start from, not as like-for-like generational peers, because “we run mini or nano today. Is a newer model on Amazon Bedrock worth it?” is the question we hear most. We focus on three questions: - What does a correct answer cost? Not what a token costs. - What do agent trajectories cost? Turn count can dominate the bill. - Can it produce work a professional would accept? This is measured on real occupational deliverables, not quiz questions. The core benchmarks here are reproducible: the harness, openai-on-aws/benchmarks-openai, runs one identical code path (the OpenAI Responses API) against both applications, and we encourage you to run it on your own tasks before making any decision. Solution overview The harness evaluates all five models through the same Responses API client, switching the backend and model ID while holding the evaluation logic constant. The results still reflect differences in the models, provider infrastructure, and model-specific configuration. In particular, the models on Amazon Bedrock ran with reasoning disabled, while the OpenAI API baselines ran at their defaults. This is a comparison of practical deployment configurations, not a controlled estimate of intrinsic model capability. We measure three things: single-call accuracy and cost on benchmarks that still separate frontier models (AIME competition mathematics, GPQA Diamond graduate-level science, and MMLU-Pro), multi-turn agent trajectories on live web-research tasks, and rubric-graded professional deliverables. Grading combines deterministic checks with a large language model (LLM) judge (gpt-5.5, not one of the evaluated models) using frozen prompts whose hashes are recorded in every result file. Each run writes a timestamped result JSON, and every number and chart in this post is generated from those files at build time. How to read the results: sample sizes range from 48–198 items. Treat small gaps as directional unless accompanied by uncertainty estimates, and reproduce the evaluation on your own workload before selecting a model. Measuring the cost of a correct answer For each benchmark we divide a model’s total spend, across right and wrong attempts alike, by its number of correct answers. This estimates the observed cost of a correct answer in our sample. The following graph shows accuracy and cost per correct answer on AIME for all five models. The pattern repeats on GPQA Diamond and MMLU-Pro (full tables in the repo). Two things stand out: - Capability tiers are visible in this sample. Sol solves 75 percent of AIME problems to mini’s 37 percent, and leads GPQA Diamond (68 percent versus 43 percent) and MMLU-Pro (82 percent versus 59 percent). If attempts were independent and accuracy remained constant, 37 percent accuracy would imply roughly 2.7 attempts per success on average. Real retries are correlated, so measure your actual retry strategy rather than assuming that estimate transfers directly. - Token efficiency decided the invoice even before prices moved, and then prices moved. At its original list price (roughly 1.5× mini’s), luna was already 25 percent cheaper per correct AIME answer in this configuration, because with reasoning disabled it used fewer billed tokens than mini at its defaults. Following the July 30, 2026 price reduction for GPT-5.6 Luna and Terra on Amazon Bedrock (luna −80 percent, terra −20 percent), our recorded cost per correct AIME answer is $0.0021 for luna versus $0.0139 for mini. Across these samples, luna has the lowest observed cost per correct answer, including against nano, even though nano’s nominal token price is slightly lower under the price assumptions recorded in the result files.Pricing note: the result files currently use $0.22/$1.32 per 1M input/output tokens for luna and $2.20/$13.20 for terra. Confirm the applicable Amazon Bedrock inference tier and region against the live pricing page before publication. The July 30 announcement describes first-party price parity, while pricing-page updates may lag.For customers, the more useful model-selection question is not only “which is cheapest per token?” but “which is cheapest per outcome, at a quality level my business accepts?” Measuring agent trajectory cost Single-call pricing misses a defining feature of our agentic workload. The harness uses client-managed history with store: false, so every turn re-sends the system prompt, prior tool results, and conversation context. Per-turn context grows roughly linearly. Cumulative billed input can therefore grow approximately quadratically with turn count. Each turn also adds a round trip of latency. A model that finishes in five turns instead of eight can save more than the 37 percent reduction in turns alone suggests. To measure this on a real workload, we ran a 50-question stratified sample of DeepSearchQA (multi-step web-research questions) through a live agent loop with real web_search and fetch_page tools. Answers were scored using a deterministic pre-pass followed by a frozen gpt-5.5 autorater. A question passes at F1 ≥ 0.7. The following graph shows how turn count drives input-token volume (left) and what each passing answer ends up costing (right). The left panel is the mechanism. The right panel is the consequence. Mini took the most turns per question of any model (7.6, mostly re-search loops), and every extra turn re-sent a context containing accumulated search results. By the end, that reached 2.3× terra’s input-token volume (114k versus 50k tokens per question). In this sample, terra’s higher token price was offset by fewer turns and higher quality: it cost $0.31 per passing answer versus mini’s $0.40, with mean F1 of 0.50 versus 0.39. The three GPT-5.6 configurations recorded higher mean F1 than both baselines, while luna combined fewer turns than mini with $0.05 per passing answer versus mini’s $0.40. Nano’s nominal token price was lower, but at an 18 percent pass rate its observed cost per passing answer was $0.07 versus luna’s $0.05. With 50 questions, treat close gaps as directional. The customer takeaway: turn efficiency is a pricing variable, and it’s invisible on the pricing page. If your agents chain tool calls (research, multi-hop lookups, iterative retrieval), benchmark trajectory cost alongside call cost. Evaluating professional deliverables with GDPval Benchmarks grade answers. Much of what customers ship is documents such as compliance briefs, financial plans, and care protocols, where “correct” is a rubric, not a string match. So we ran a 48-task slice of GDPval: real occupational deliverables created by professionals averaging 14 years of experience, each graded against its human-authored rubric. A deliverable passes at ≥70% of weighted rubric points. The following graph shows pass rates and cost per passing deliverable. All three gpt-5.6 configurations achieved higher observed rubric scores than mini and nano with reasoning disabled. The largest observed differences appeared in law, nursing, and financial-advice tasks, where rubrics demand specific caveats, structure, and completeness. The per-category samples are small, so treat this pattern as exploratory. Luna scored higher than mini on 31 of 48 deliverables, lower on 9, and tied on 8. It passed 27 deliverables to mini’s 20. This workload is a single call with no turn effects, so before the repricing, mini and nano were cheaper per passing deliverable and gpt-5.6 bought pass rate alone. After repricing, luna has the lowest observed cost in this sample: $0.010 per passing deliverable versus mini’s $0.030 and nano’s $0.012, at a 56 percent pass rate to mini’s 42 percent and nano’s 35 percent. Terra and sol still carry a premium per pass while recording higher rubric performance. Whether that premium is justified depends on the cost of review and rework in the target workflow. GDPval caveat: outputs were capped at 8,192 tokens, truncating 6 luna, 9 terra, 7 sol, 0 mini, and 1 nano deliverables. These are real outcomes under the tested cap. A higher cap may improve quality, but it might also increase cost. Test both together. Putting it together: a decision framework If you run gpt-5.4-mini or nano today, the migration question comes down to workload shape: We also measured same-model latency on both applications with the same streaming harness. In our July 2026 runs (us-west-2, single-region, and point-in-time), median time-to-first-token on Amazon Bedrock averaged 21 percent lower for luna and 5 percent lower for terra across the 12 matched configurations. Shared services vary with load, so treat these as a snapshot and re-measure. Luna’s throughput averaged 43 percent higher on Amazon Bedrock at ≥500-token outputs (terra +4 percent). The observed worst-case TTFT-to-median ratios were 2.1–2.5× on Amazon Bedrock versus 4.6–6.6× on the OpenAI API. These maxima indicate tail variability but are not estimates of p99 latency. Sol behaves differently: it’s a deep-reasoning model with inherently long and variable time-to-first-token, and its Amazon Bedrock runs used us-east-1. Per-configuration detail for all three models is documented in the repository’s performance section, and performance/run_all.sh reproduces the comparison on your own account. Run it on your workload These results reflect our sample sizes and configuration choices: 50–198 questions per suite (DeepSearchQA 50, AIME 60, MMLU-Pro 140, GPQA Diamond 198), 48 deliverables, and reasoning disabled for the models on Amazon Bedrock, a deliberate cost floor. Enabling it raises both quality and spend. Your tasks are not our tasks. The methodology is the durable part: Swap in 50–100 of your own tasks with known-good outputs and the same scripts produce cost-per-success numbers in your domain, on your account. Conclusion The per-token price is one input to a decision, not the decision. In the configurations we tested, normalizing cost by successful outcomes changed the model ranking after accuracy, token efficiency, and trajectory length were included. Pricing pages also move, as the July 2026 GPT-5.6 reductions showed. Define the outcome and quality threshold your application needs, measure the full cost of successes and failures, and re-run the evaluation when prices, models, or workload patterns change. To get started with OpenAI models on Amazon Bedrock, see the Amazon Bedrock documentation. To reproduce these benchmarks on your own tasks, clone openai-on-aws/benchmarks-openai and follow the steps in the previous section.
document fetched 2026-09-12 02:56 · canonical https://aws.amazon.com/blogs/machine-learning/beyond-the-price-per-token-choosing-the-right-openai-model-on-amazon-bedrock-for-your-workload