loading
loading
Artificial Analysis' Code2Video-AA Benchmark Shows Claude Opus 5.5 Dominates AI Video Coding A new benchmark judges 27 models on turning marketing briefs into animated HTML videos, with Claude Opus 5.5 leading at 10x the cost of rivals. - Artificial Analysis launched Code2Video-AA, benchmarking 27 models on 114 marketing video briefs rendered from HTML. - Claude Opus 5.5 leads at 1536 Elo, winning all five judge axes including craft and engagement. - Cost per task spans 300x, from $0.01 for MiMo-V2.6-Flash to $3.89 for Claude Fable 5.1. - GPT-6.1 Sol delivers 1369 Elo at $0.31 per task, roughly 10% of Opus's cost. - Open weights models trail the frontier by nearly 500 Elo points on this visual design workload. - Over 35,000 pairwise verdicts from a Gemini 4 Argon judge produced the Elo rankings with 95% confidence intervals. Artificial Analysis has released Code2Video-AA, an evaluation developed with HeyGen that tests whether language models can translate marketing briefs into animated videos using code alone. Conventional coding and reasoning benchmarks reveal little about layout, motion, timing, and visual polish. This benchmark gives teams building code-driven video products a shared comparison of output quality, render reliability, and generation cost. Artificial Analysis runs Code2Video-AA as an independent implementation of HeyGen’s Code2Video Bench. Across 114 briefs, each of 27 models creates a HyperFrames composition in a single HTML file. HTML defines the layout, GSAP controls the animation, and HyperFrames renders the result as video. One file, one render - Prompt: The evaluator sends a design brief and HeyGen’s system prompt in one model call, with tools disabled. - Generation: The model returns a single index.html file containing the complete composition. - Rendering: HyperFrames renders the file offline with no network access. Missing CDN assets, unavailable scripts, and invalid code can cause the submission to fail. - Judging: A failed render loses every comparison against a successfully rendered video for the same brief. Gemini 4 Argon compares pairs of rendered videos without seeing the model names. It selects a winner on five axes: engagement, prompt intent, composition, temporal consistency, and craft. A Bradley-Terry model converts those pairwise verdicts into an Elo rating for each axis, and the benchmark averages the five ratings for its overall score. GPT-5.5 anchors the scale at 1,000, so every Elo score is relative to performance within this evaluation. Claude takes all five axes Claude Opus 5.5 leads with 1,536 Elo and ranks first on every judging axis. Claude Sonnet 5.5 follows at 1,443, while GPT-6 Astra scores 1,427; the benchmark reports those two models as statistically tied for second. GPT-6.1 Sol places fourth at 1,369. Opus records its largest lead on craft, where it finishes 128 points above Sonnet. Its narrowest lead is 48 points on prompt intent. The leading open-weight models remain nearly 500 points behind Opus: DeepSeek V4.1 Flash scores 1,053, while MiMo-V2.6-Flash and MiMo-V2.6-Pro each score 1,052. Costs span more than 300-fold Reported cost per task ranges from $0.01 for MiMo-V2.6-Flash to $3.89 for Claude Fable 5.1. The same benchmark workload therefore carries a price spread exceeding 300-fold across providers and models. Eight models sit on the Elo-to-cost Pareto frontier, meaning no alternative in the evaluation offers both a higher score and a lower reported cost. Claude Haiku 5.5 reaches 1,192 Elo at $0.08 per task, while GPT-6.1 Sol reaches 1,369 at $0.31. GPT-6 Astra gains 58 Elo over Sol but costs $1.62 per task, about 5.2 times as much. Reasoning tokens account for most of the variation in output usage. Average output ranges from about 11,000 tokens per task for Nemotron 3 Ultra to about 176,000 for Claude Sonnet 5.5, while the final HTML compositions occupy roughly 4,000 to 12,000 tokens. The three Claude 5.5 models use approximately 152,000 to 176,000 output tokens per task, compared with a median of about 34,000 across the other 24 models. One score bundles three skills A high Code2Video-AA score requires models to satisfy dense instructions, generate executable front-end code, and make coherent visual choices within one response. - Long-form instruction following: Briefs specify details such as hex colors, scene order, animation timing, and camera behavior. - Front-end code generation: Models must produce a working GSAP animation in one HTML file without a debugging loop. - Visual judgment: Engagement and craft reward hierarchy, pacing, polish, and other qualities that extend beyond technical correctness. Rendering failures explain little of the separation among the strongest models. Of 3,078 submissions, 3,042 rendered successfully, a rate of 98.8%. Eight models completed all 114 briefs, and the lowest result was GLM-5.3 with 110 successful renders. Among models with near-perfect reliability, Elo differences mainly reflect the judge model’s visual preferences. A narrow slice of visual development Code2Video-AA measures a specific workflow: single-call generation of marketing animations in HyperFrames. Several boundaries affect how its results should be applied. - No iteration: The setup excludes tool use, debugging, human feedback, and multi-step agent workflows. - One rendering stack: Results cover HTML and GSAP compositions rendered through HyperFrames. Other animation frameworks and video-editing systems may produce different rankings. - One automated judge: Gemini 4 Argon supplies every aesthetic verdict, so its stylistic preferences influence the Elo ratings. - Relative ratings: Elo supports comparisons among models in this benchmark. It does not provide an absolute percentage of visual quality or production readiness. Sol narrows the gap at $0.31 Teams building code-driven motion products can use the Pareto frontier to identify candidates for further testing. GPT-6.1 Sol trails Opus by 167 Elo while costing $0.31 per task, and Claude Haiku 5.5 offers a lower-cost option at $0.08. GPT-6 Astra provides a smaller score gain over Sol at a substantially higher reported price. Opus remains the strongest candidate when judge-rated craft is the primary selection criterion. A production evaluation should also cover representative brand briefs, browser compatibility, generation latency, editing requirements, and failure recovery. Those factors fall outside Code2Video-AA but can materially change the cost of deploying a model. The full leaderboard includes model-level results and rendered samples. Artificial Analysis also publishes its methodology, while HeyGen provides the HyperFrames renderer on GitHub for local reproduction.
document fetched 2026-10-28 21:03