loading
loading
Epoch Finds Half of 21 Major AI Benchmarks Are Flawed Epoch AI expanded its benchmark audit program, flagging popular evals like SWE-bench Verified and Humanity's Last Exam as Flawed under a public rubric. - Epoch AI released 8 new benchmark reviews: 1 Verified, 3 Flawed, 4 Not Enough Info - Rubric flags benchmarks as Flawed when scoring defects affect at least 20% of the score - SWE-bench Verified, SWE-Bench Pro, Terminal-Bench, HLE, and HealthBench Professional all rated Flawed - GraphWalks, ExploitBench, SimpleQA Verified, PostTrainBench, and WeirdML v2 earned Verified status - Full benchmark list and methodology are public - Reviewer capacity is the main bottleneck; Epoch is hiring external experts Epoch AI has added eight audits to its public Benchmark Reviews program. One new entry earned a Verified rating, three were rated Flawed, and four lacked enough information for a verdict. Across all 21 reviews, 10 benchmarks are now rated Flawed. Benchmark scores shape model selection, research claims, and training decisions. Errors in tasks, scorers, or evaluation settings can distort those scores, so Epoch audits the underlying materials rather than relying on leaderboard results alone. Inside the 20% cutoff Epoch first determines whether reviewers can inspect the benchmark’s tasks, scoring logic, evaluation harness, and model settings. Required settings include reasoning effort, token and time limits, tool access, API configuration, and system prompts. Benchmark owners may provide private materials under a nondisclosure agreement. - Verified - Reviewers found no scoring defect that could affect at least 20% of the benchmark’s full score. The published review may still document limitations and weaker design choices. - Flawed - One or more scoring defects could affect at least 20% of the full score. - Not Enough Info - Reviewers could not access enough tasks, scoring logic, harness code, or model settings to complete the audit. Epoch’s scoring audit looks for defects that can inflate or suppress reported performance: - Impossible tasks caused by missing files, hidden requirements, or underspecified prompts - False negatives caused by overly strict scorers, stale reference answers, or sandbox failures unrelated to the model - False positives caused by permissive scorers or reward hacks, including shortcuts that retrieve answers from the harness or the web - Elicitation failures that deny a model the tokens, turns, time, or tools needed to demonstrate its capabilities - Model-specific scaffolds that give some systems an uneven advantage For benchmarks containing more than 50 tasks, reviewers inspect a random sample of 50 stratified by category. They expand the sample to 100 when the observed error rate falls between 15% and 25%, the range surrounding the Flawed threshold. Ten of 21 are flawed Each verdict applies to the benchmark version and evaluation setup Epoch reviewed. Teams can still use a Flawed benchmark after checking whether the documented defects affect the capability or comparison they care about. Epoch publishes a limited account of identified flaws and a fuller report for Verified benchmarks, including known weaknesses and constraints. How to use the verdicts - Check the verdict, benchmark version, and evaluation configuration before citing a leaderboard score in a paper, model card, or report. - For internal regression testing, prefer Verified benchmarks or confirm that documented flaws do not affect the capability under test. - Treat results marked Not Enough Info as unaudited until the required tasks, scorers, harness code, and settings become available. - When comparing models, match token budgets, time limits, reasoning settings, tools, prompts, and scaffold behavior. - Benchmark maintainers can use Epoch’s methodology as an audit checklist. Disclosed harnesses, versioned scorers, and human-completability data make defects easier to find and reproduce. Review capacity sets the pace Epoch says reviewer capacity limits the program’s expansion. The group plans to prioritize benchmarks using three criteria: - Impact: coverage of widely used or safety-relevant capabilities - Reach: citation frequency and inclusion in recent system cards - Diversity: coverage of domains and capabilities missing from the current review set Benchmarks in fields such as physics and medicine take longer to assess because the audits require outside specialists. Access restrictions can also produce a Not Enough Info verdict even when a benchmark is widely used. To avoid conflicts of interest, Epoch excludes its own benchmarks from the program and may commission external reviews of them later. The group cites a prior audit that found errors in 42% of FrontierMath v1 problems, showing why benchmark creators also need independent scrutiny. Developers and researchers can use these reviews as an evidence layer when selecting evaluations, comparing systems, or assessing published claims. Epoch is also advertising an open position to expand the program’s review capacity.
document fetched 2026-10-28 15:05