Cartesia's Managed Agents Beats Gemini and Grok With 91.7% Task Success

retained text

Cartesia's Managed Agents Beats Gemini and Grok With 91.7% Task Success
Cartesia's new Managed Agents platform jumps 98 Elo points over its predecessor on blind voice preference tests, with task success climbing to 91.7%.
- Cartesia Managed Agents hits Speech Agent Arena Elo of 1115, up 98 from Line.
- Task Success Rate on tool-calling conversations jumps from 77.5% to 91.7%.
- Cascaded system: Cartesia's Ink STT and Sonic TTS wrapped around a developer-chosen LLM.
- Evaluated using GPT-5.5 plus the Skylar and Daniel voices, 685 samples.
- Pricing is $0.06 per call minute, with LLM usage billed separately.
- Beats native speech-to-speech models from Google and xAI on preference Elo.
Cartesia Managed Agents posted an overall Elo rating of 1115 on the Speech Agent Arena, 98 points above the company’s earlier Line framework at 1017. The systems’ 95% confidence intervals do not overlap. On conversations requiring tool calls, Task Success Rate rose from 77.5% to 91.7%.
The result gives developers evidence that a hosted pipeline combining speech recognition, a selectable large language model, and speech synthesis can compete with native speech-to-speech systems. It also reflects one benchmark configuration, so the scores cannot identify which component produced the improvement.
Inside the blind calls
The Speech Agent Arena is a blind, head-to-head benchmark for live voice systems. A participant calls two unidentified agents separately using the same scenario, then records an overall preference and answers diagnostic questions about the conversations.
Elo converts those preference votes into a relative rating. A higher score indicates stronger performance against the systems included in the arena, while the confidence interval represents uncertainty around that estimate. Ratings can change as the benchmark adds votes and models.
The benchmark divides its scenarios into two groups:
- Agentic: 15 tasks that require tool calls, such as ordering takeout or booking an appointment.
- Non-agentic: 20 informational tasks, such as asking a business about its opening hours.
Task Success Rate applies to eligible agentic conversations. The metric counts whether the system completed the requested action through the correct final tool call or sequence of calls, independent of the participant’s preference vote.
In a dental-booking scenario, for example, the agent must retrieve the participant’s account with get_account_details, query open appointments with check_availability, and complete the reservation with book_slot. A fluent conversation that omits the final booking call counts as a failed task.
A managed cascade, component by component
Managed Agents documentation describes a cascaded voice stack. Cartesia’s Ink model transcribes incoming speech, a developer-selected LLM decides how to respond or use tools, and Sonic converts the response into audio. The benchmark configuration used GPT-5.5 with the Skylar and Daniel voices.
- Speech input: Ink produces streaming transcripts and detects conversational turns.
- Reasoning and tools: The selected LLM follows instructions and chooses when to call configured tools.
- Speech output: Sonic synthesizes and streams the spoken response.
- Call orchestration: Cartesia manages telephony, turn-taking, and communication among the components.
Developers configure the agent’s instructions, voice, LLM, and available tools instead of building the call-orchestration layer themselves. Application-specific operations still require tools that expose the relevant business logic.
Managed Agents succeeds Cartesia Line, a code-first framework for voice applications. The hosted product moves more infrastructure into Cartesia’s platform, reducing the amount of telephony, turn-taking, provider coordination, and tool wiring that development teams must operate.
The numbers behind the lead
At the cited leaderboard snapshot, Cartesia’s 1115 Elo placed it above Gemini 3.1 Flash Live Minimal at 1100, Gemini 3.8 Live at 1088, and Grok Voice 3.0 at 1070. Grok completed 94.8% of eligible tasks, 3.1 percentage points above Managed Agents despite receiving a lower preference rating.
How to interpret the result
Native speech-to-speech models process audio directly, which can preserve prosody, emotion, and other vocal signals that a transcript may discard. Cascaded systems introduce separate transcription, reasoning, and synthesis stages, creating more opportunities for latency and coordination errors.
Cartesia reduces those costs through streaming components. Sonic begins streaming audio in about 90 milliseconds, while Ink provides streaming transcription and native turn detection. The architecture also lets developers replace the reasoning model without retraining the speech layers, allowing the stack to adopt newer LLMs as they become available.
The comparison changes several variables at once, including the speech components, LLM, orchestration layer, and product architecture. An ablation study that swaps one component at a time would be required to attribute the 98-point Elo increase or the higher completion rate to a specific change.
Developers evaluating the stack can use the arena’s benchmark methodology and results as evidence from controlled, human-tested calls. Production testing still needs to cover workload-specific factors such as latency distribution, interruptions, accents, background noise, tool failures, compliance requirements, and total cost.

document fetched 2026-10-28 16:03