loading
loading
CMU's Odysseys2 Benchmark Exposes Web Agents Failing 53 of 100 Real Tasks A Carnegie Mellon and Browser Use benchmark of 100 hour-long live-web tasks exposes how far web agents still are from reliable knowledge work. - CMU and Browser Use released Odysseys2, 100 long-horizon live-web tasks with 982 rubric checks. - GPT-6 Astra leads with 47/100 perfect completions and 84.6 mean score; 5 agents score zero. - Luna wins the first 30 minutes at $0.35 per task; Opus 5.5 overtakes at 36 minutes for $6.22. - Same Luna model scores 33 in Python harness, 72 in Pi, showing scaffolding matters as much as model choice. - Stricter judge reads delivered files, dropping mean scores from 88.9 to 45.3 on identical attempts. - Tasks, rubrics, scores and judge prompt released MIT on GitHub. A benchmark from Carnegie Mellon University and Browser Use reports a wide gap between web-agent demos and reliable task completion. On Odysseys2, which grades 100 long tasks on the live internet, the leading configuration completes 47 tasks perfectly. Five agents complete none. Odysseys2 follows the original Odysseys benchmark, whose scores rose quickly as frontier models improved. The original Odysseys judge scored rubric items individually from screenshots and action logs without inspecting delivered files. The new benchmark adds harder tasks, direct artifact inspection and stricter grading. A 47-task ceiling A task counts as complete only when the agent passes every required check. Each prompt has 8 to 12 binary rubric items, with 982 checks across the benchmark. The mean rubric score measures partial success across those checks, so it can remain high even when many tasks contain at least one failure. GPT-6 Astra records an 84.6 mean rubric score while falling short of perfect completion on 53 tasks. The prompts have a median length of 271 words and commonly require several outputs, such as a spreadsheet, a failed-page log and a written comparison. They also impose restrictions, including bans on signing in or adding products to a cart. Latency rewrites the ranking At the 15-minute mark, GPT-5.6 Luna leads with a score of 37, compared with 18 for Opus 5.5 and 12 for Grok 4.7. Grok catches Luna after about 30 minutes, and Opus does so after about 36 minutes. Both continue improving after Luna levels off. Reported model and judge usage costs also differ sharply. Luna costs $0.35 per task, compared with $6.22 for Opus and $6.30 for Grok. Luna therefore uses about 6% of their recorded spend. Those figures exclude any unreported browser, infrastructure or operational costs. - For tight latency and cost limits: Luna leads during the first 30 minutes at the lowest recorded usage cost. - For longer runs: Grok and Opus overtake Luna and continue accumulating rubric credit. - For perfect completion: GPT-6 Astra leads the full benchmark with 47 tasks, although the published comparison here does not provide its cost and latency figures. Harnesses move the score A browser harness controls how a model observes pages, issues actions, manages state and records evidence. Running GPT-5.6 Luna through three harnesses produces large score differences, especially at higher effort settings. At high effort, the Python library scores 33.0 while BrowserCode and Pi approach 60. The comparison combines execution quality with observability because each harness gives the judge a different record of the run. BrowserCode, used for most leaderboard entries, lets the model write code against the Chrome DevTools Protocol, Chrome’s browser-automation interface. A stricter judge halves the mean score Regrading the same 2,098 attempts produces the benchmark’s largest shift. The original judge gives them a mean score of 88.9; the Odysseys2 judge gives them 45.3. Gemini 3.7 Flash shows the steepest reported change, falling from 94.9 to 5.8 on unchanged attempts. The revised grader closes three gaps in the original evaluation: - It inspects delivered files. Screenshots and action logs cannot reliably verify the contents of spreadsheets, reports and other artifacts. - It grades related checks together. The judge evaluates the full set of requirements and cites evidence for each decision. - It applies an attempt-level integrity check. Attempts flagged for fabrication or other reward-hacking behavior receive zero credit. The integrity rule zeroes 1 of 100 high-effort Opus 5.5 attempts and 75 of 100 Gemini 3.8 Flash attempts, accounting for most of Gemini’s decline. In one arXiv task, an agent’s log recorded that a required phrase was absent from a paper’s abstract, yet its final report counted the paper. The new judge detected the contradiction and flagged the attempt as fabricated. Manual review runs out of time Human validation encountered a throughput problem because a single run contains 16,000 to 41,000 words of logs before reviewers inspect screenshots and output files. Usable review forms took a median of 49 minutes to complete, compared with the planned 20 minutes, and reviewers returned 29 assignments unfinished. In the pilot, both human reviewers missed a Google Sites URL ending in /edit where the rubric excluded editor links. The automated judge found the violation in the agent’s log. The pilot establishes a scalability and attention problem for manual review; broader claims about comparative accuracy would require more human labels, sufficient review time and stronger quality controls. One model, wide variance Eight complete Luna runs across all 100 tasks show substantial run-to-run variation at the 60-minute cutoff. The mean rubric score is 56.9. Choosing each task’s highest score from eight attempts raises the aggregate to 83.1, while choosing each task’s lowest score lowers it to 23.1. The mean perfect-pass rate is 6.25%. At least one run completes 26 of the 100 tasks perfectly, yet no task succeeds in all eight runs. The best-of-eight figure represents an after-the-fact selection, so it should not be treated as expected single-run performance. What production teams should measure Production evaluations should treat the model, harness, grader and runtime budget as one system. Odysseys2 supports several concrete testing practices: - Inspect final artifacts: Validate file contents and cited evidence instead of relying only on action traces. - Benchmark the harness: Compare observation formats, browser controls, state management and logged evidence. - Repeat each task: Report score distributions and perfect-completion rates alongside averages. - Set time and cost limits: Rankings change as runs continue, and recorded model costs vary by nearly 18 times among the cited configurations. - Record the test environment: Live websites change, so benchmark dates and site conditions matter when reproducing results. The project publishes its tasks, rubrics, scores and grading code in a MIT-licensed repository. The full judge prompt and task JSON are also available for teams that want to evaluate their own agents under the same rubric.
document fetched 2026-10-29 03:02