Artificial Analysis Ranks Claude Opus 5.5 First in Tougher Coding Agent Index

retained text

Artificial Analysis Ranks Claude Opus 5.5 First in Tougher Coding Agent Index
A new coding agent leaderboard swaps in Cognition's FrontierCode benchmark and tightens Q&A grading, reshuffling the top of the charts.
- Artificial Analysis launched Coding Agent Index v2.0 across 290 tasks spanning SWE, terminal, and repo Q&A.
- FrontierCode v1.1 from Cognition replaces DeepSWE, grading on whether a maintainer would merge the change.
- Claude Opus 5.5 in Claude Code leads at 61.7, with 56.0% on FrontierCode specifically.
- Mean scores dropped from 54.8 to 47.4 across 21 shared configs due to stricter grading.
- Cost frontier spans $0.39/task (DeepSeek V4.1 Flash) up to $11.42/task (Opus 5.5).
- SWE-Atlas-QnA-AA dataset is public on Hugging Face.
Artificial Analysis has released version 2.0 of its Coding Agent Index, with Claude Opus 5.5 running in Claude Code leading at 61.7. The release introduces a harder repository benchmark and stricter Q&A judging. Among the 21 configurations evaluated under both versions, 20 received lower composite scores.
A tougher yardstick reshapes the index
The index scores a model together with its agent harness, the software layer that controls prompts, tools, commands, and iteration. Version 2.0 covers 290 tasks across three equally weighted benchmarks, so each benchmark contributes one-third of the composite score regardless of its task count.
FrontierCode was created by Cognition with maintainers of open-source projects. Its tasks ask an agent to modify a real repository, then assess whether the resulting patch meets a maintainer’s merge standard. The 100-task main set remains private to reduce training-data contamination. That secrecy also limits independent inspection of individual tasks.
SWE-Atlas-QnA-AA retains Scale AI’s 124 repository questions. Artificial Analysis reworded 127 of the 1,306 rubric criteria for consistency and moved judging to GPT-6 Luna with the “xhigh” setting. Five judging votes are cast for every criterion. The dataset and judge files are available on Hugging Face.
Mergeability becomes the bar
FrontierCode evaluates each attempt with six classes of checks: classical tests, command checks, reverse-classical tests, adaptive classical tests, scope checks, and LLM prompt review. Together, they assess program behavior, command execution, patch boundaries, and issues requiring semantic review.
Every check is classified as blocking or non-blocking. One failed blocking check reduces the attempt’s score to zero. Artificial Analysis also runs a modified version of Cognition’s rule-based trajectory scanner; attempts flagged for accessing solution material receive zero.
Attempts that pass every blocking check earn between 70% and 100%, depending on their non-blocking results. Artificial Analysis runs each task three times through the Stirrup harness, allows up to 250 turns per run, and weights every task equally.
Opus 5.5 leads a compressed field
Claude Opus 5.5 holds the highest composite score and the strongest reported FrontierCode result among the selected configurations. Several vendors did not report FrontierCode or cost figures for every entry.
Opus 5.5’s advantage comes mainly from FrontierCode, where it scores 56.0%. The broader field is tightly grouped: 15 of the index’s 24 configurations score between 43% and 49% on that benchmark.
SWE-Atlas-QnA-AA produces a different order. Muse Code with Muse Wonder Full leads at 68.8%, followed by Claude Code with Fable 5.1 at 66.4% and Claude Code with Opus 5.5 at 65.9%.
New rules pull scores down
Across the 21 configurations tested on both index versions, the mean composite score fell from 54.8 to 47.4. The benchmark replacement accounts for most of the decline: those configurations averaged 45.1% on FrontierCode v1.1 and 65.6% on the retired DeepSWE v1.1.
The revised Q&A judging reduced the average by another 1.7 points. Individual configurations moved by as much as seven points in either direction, showing that the new judge also changed relative performance.
Because the task set and judging method changed, the decline measures methodology alongside agent capability. Scores from versions 1.0 and 2.0 are not directly comparable. Comparisons within version 2.0 place every configuration under the same rules.
Price changes the ranking
Among configurations with published cost estimates, four sit on the cost-performance frontier: Codex with DeepSeek V4.1 Flash, GPT-6.1 Sol, GPT-6 Astra, and Claude Code with Opus 5.5. A configuration belongs to this frontier when no measured alternative provides both a higher score and a lower cost.
DeepSeek V4.1 Flash offers the lowest reported price at $0.39 per task and scores 47.3. Sol raises the score to 54.1 for $1.50, Astra reaches 54.7 for $7.04, and Opus 5.5 reaches 61.7 for $11.42.
These estimates use pay-per-token API prices. Claude Code and Codex Pro subscriptions follow different pricing models, so subscription-based deployments require a separate cost calculation.
Match the benchmark to the job
Earlier coding benchmarks, including SWE-bench and DeepSWE, often produced results in the 60% to 70% range under different tasks and grading rules. FrontierCode applies a more selective standard through private repository tasks, blocking checks, scope review, and trajectory scanning. Its scores measure success under those controls; they do not establish production reliability on every codebase.
A team evaluating pull-request automation should give FrontierCode more weight than the composite score. Repository explanation work aligns more closely with SWE-Atlas-QnA-AA, while shell automation maps to Terminal-Bench. The equal-weight composite can obscure those workload-specific differences.
Artificial Analysis publishes its methodology version history and detailed FrontierCode results. Before deployment, teams can pair those results with a small evaluation drawn from their own repositories, review standards, tools, and task mix.

document fetched 2026-10-23 22:18