loading
loading
Vals AI Finds the Same Open-Weight Model Costs 2.3x More Depending on Provider Vals AI's new Inference Provider Index shows the same open-weight model can cost 2.3x more and run 5x slower depending on where you host it. - Vals AI launched the Inference Provider Index comparing open-weight model endpoints on agentic tasks. - Same model cost up to 2.3x more depending on provider, with model-maker APIs never cheapest or fastest. - Kimi K3 on Baseten cost $12.65/task vs $5.43 elsewhere due to lower cache hit rate (72% vs 94%). - Accuracy agreed in aggregate but split by domain: Kimi K3 Legal dropped from 75.8% to 60.6% between providers. - Open-weight models crossed 62% of OpenRouter tokens since July, making provider choice architecturally critical. - Benchmark reruns every two weeks via open-source Valkyrie harness to catch silent regressions. Vals AI has released the Inference Provider Index, which compares model creators’ APIs with third-party endpoints for DeepSeek V4.1 Flash, Kimi K3, and MiniMax M3. Across agentic finance, coding, and legal tasks, the same model cost up to 2.3× more depending on its provider. For every model, at least one third-party endpoint charged less and at least one generated tokens faster than the creator’s API. Open-weight releases allow several companies to host the same model, while each provider controls caching, numerical precision, inference kernels, decoding, prompt templates, and tool-call parsing. Production behavior reflects both the released weights and that serving stack, so posted token prices alone cannot predict task cost, speed, or accuracy. Cache hit rates drive the 2.3× gap Kimi K3 cost $12.65 per task through Baseten, compared with $5.43 to $5.97 through the other tested providers, despite producing roughly the same output length. Baseten served 72% of Kimi K3’s input from cache, while Together AI and Fireworks each reached 94%. On coding tasks, the corresponding rates were 70% and 98%. Input caching reuses prompt prefixes that a provider has already processed, which matters for agents that repeatedly send instructions, tool definitions, and conversation history. Providers usually bill cached input below the full input-token rate. A lower hit rate therefore pushes more of each request onto the higher rate, even when listed token prices appear similar. MiniMax M3 showed the reverse pricing pattern. It cost $1.01 per task through Together AI and $1.07 through Fireworks, compared with $2.18 through MiniMax’s API. The creator endpoint remains a useful baseline, while measured task economics favor the third-party options in this test. Domain splits expose accuracy gaps Aggregate accuracy differences generally fell within the benchmark’s error bars, which reflect uncertainty from a finite task sample. Domain-level results varied more sharply. Kimi K3 scored 60.6% on legal tasks through Baseten and 75.8% through Moonshot’s API, a 15.2 percentage-point gap. On coding tasks, Fireworks reached 87.7%, compared with 80.1% through Moonshot. Providers can quantize weights, storing them with fewer bits to reduce memory use, and can deploy different kernels, decoding settings, request templates, or tool parsers. The resulting scores characterize the complete hosted endpoint, including those implementation choices. Earlier comparisons of gpt-oss-120b found double-digit provider gaps that narrowed after vendors corrected their deployments. Model creators have also started publishing validation tools; Moonshot provides a Kimi vendor verifier for checking hosted implementations. Throughput moves independently of price Kimi K3 generated a median 229 tokens per second through Baseten, about five times Moonshot’s 45 tokens per second. Baseten combined the model’s highest measured task cost with its fastest generation rate. DeepSeek V4.1 Flash showed a different ordering. Fireworks charged $0.36 per task and generated 103 tokens per second, while DeepSeek’s API charged $0.55 and reached 286 tokens per second. Together AI delivered the highest median generation throughput in the comparison while charging less than the creator endpoint. 165 tasks across three agent workflows Vals evaluates multi-step agents on 165 tasks drawn from three existing benchmark suites. Each task can involve several model requests, tool calls, and intermediate decisions, making the test closer to an agent workflow than a single-turn question. - Finance Agent v2: 66 financial-analysis tasks using company filings. - Vibe Code Bench: 33 tasks that require building web applications from a specification. - Legal Research: 66 research and question-answering tasks involving US law. Requests pass through Vals’ open-source model library, which records timing, token usage, and cost for each call. The index aggregates those requests into per-task figures. Vals runs the agents with Valkyrie, its open-source agent harness. Vals reruns the index every two weeks and establishes a fresh baseline through each model creator’s API. That schedule accounts for provider changes to kernels, quantization, caching, and speculative decoding that can alter results without changing the advertised model name. Provider rankings change with deployment The index covers three models, three domains, and 165 tasks, so its rankings apply to those workloads and the endpoints tested during each run. Applications with different prompt lengths, tool schemas, output formats, or domain requirements can produce another ordering. Generation throughput describes how quickly an endpoint emits output tokens once generation is underway. Production evaluations also need time to first token, end-to-end task duration, error rates, retries, rate limits, and structured-output reliability. Open-weight models carry most router traffic OpenRouter routes model requests among multiple inference providers, making provider-level differences relevant across a growing share of deployed workloads. Vals reports that open-weight models accounted for more than half of OpenRouter tokens since April and more than 62% since July. Benchmark the endpoint you will ship - Use representative tasks. Run the same prompts, tool schemas, context lengths, and decoding settings expected in production. Aggregate scores can conceal domain gaps exceeding 15 percentage points. - Calculate effective task cost. Track uncached input, cached input, output tokens, retries, and failed calls. Listed token rates omit differences in cache utilization. - Measure latency components separately. Record time to first token, generation throughput, and full task duration alongside quality. - Keep a creator baseline. Compare the model maker’s API with at least two third-party endpoints using the same harness and scoring method. - Repeat the evaluation. Rerun tests after provider updates and on a regular schedule because serving stacks can change behind stable model names. An agent that makes dozens of model calls magnifies each provider difference. At fixed volume, a 2× task-cost multiplier doubles inference spend, while a 5× throughput gap reduces the generation phase to roughly one-fifth for equal-length outputs. Provider testing therefore belongs in ongoing capacity, reliability, and cost planning.
document fetched 2026-10-27 05:33