loading
loading
OpenAI's GPT-6 Astra Tops Physics Benchmark With 83.1% on CritPt v2 The revised physics research benchmark gives models a code sandbox, and frontier systems have jumped from single-digit to 80%+ accuracy. - CritPt v2 launches with 70 revised research-level physics challenges and agentic code execution. - GPT-6 Astra (max) tops the leaderboard at 83%, GPT-6.1 Sol (max) at 82%. - Claude Opus 5.5 reaches 77.7%; GPT-5 (high) sits at 29%, a 54-point gap. - Authors fixed underspecified problems, typos, and grader stability after community feedback and expert audits. - Models run inside an offline Linux sandbox with a Python code-execution tool for computation. - Grading API remains free for approved labs via critpt@artificialanalysis.ai; paper here. The revised CritPt v2 physics benchmark has a new leader: GPT-6 Astra scored 83.1% under an agentic evaluation that lets models write and execute code. The same model scored 32% on v1. That increase combines a repaired dataset, code access, and model improvements, which prevents a direct comparison between the two results. A hard test for research-style physics CritPt was created by working physicists to test whether language models can handle tasks expected of entry-level research assistants. The release describes 71 composite challenges, with results reported over a 70-challenge test set, plus 190 modular checkpoints that isolate intermediate skills. Test solutions remain private to limit training-data contamination. Artificial Analysis initially reported Gemini 3 Pro Preview as the strongest no-tool model on v1 at 9.1%. In the original paper, GPT-5 at its high reasoning setting led the tested no-tool configurations at 5.7%, while the other no-tool models scored 2% or less. A later v1 evaluation placed GPT-6 Astra at 32%. Repairs and a sandbox reset the baseline Version 2 corrects underspecified prompts, factual errors, and typographical mistakes across the challenge set. An independent audit of 56 original challenges found benchmark errors in 21, indicating that a substantial share of v1 difficulty came from defective questions. The evaluation also gives each model a code-execution tool inside an offline Linux sandbox. Models can use Python for symbolic algebra, numerical arrays, simulations, and answer generation while remaining disconnected from the web. Artificial Analysis attributes the larger share of the score increase to the revised challenges, with tool access contributing additional gains. Because v2 changes both the questions and the permitted workflow, score differences from v1 cannot isolate model progress. Four systems clear 75% Leading models now occupy an eight-point range, with four configurations scoring at least 75%. Configuration labels below reproduce those reported by Artificial Analysis; “max” denotes the highest reasoning setting used in the evaluation. The 54.1-point gap between GPT-5 and GPT-6 Astra shows that v2 still separates older and newer systems. The comparison measures performance under the revised agentic setup, not the pace of model improvement across benchmark versions. Project-sized tasks, exact grading Each challenge comes from a researcher’s specialty and covers one of 11 modern physics subfields, with condensed matter representing the largest share. The composite problems resemble small research projects, while the modular checkpoints test individual steps within those projects. Problems use unpublished prompts designed to resist web search and memorization. Answers often take the form of symbolic expressions, numerical outputs, or executable Python functions, reducing the chance of success through guessing. A machine-verifiable grading pipeline evaluates those outputs without relying on an LLM judge, which can misread specialized notation and advanced physics formats. Reliability remains a separate concern. The original paper counted a challenge as consistently solved when a model answered correctly in at least four of five runs. GPT-5 at high reasoning met that threshold on three of 70 challenges, while the other reasoning-focused models solved none consistently. The published v2 headline scores report accuracy without equivalent run-to-run consistency results. Free access, private answers Researchers can request free grading API access through critpt@artificialanalysis.ai. Artificial Analysis and the CritPt team approve laboratories and researchers individually, following the same access model used for v1. The Hugging Face dataset contains the public benchmark material, while solutions for the 70 test challenges remain private. That arrangement allows independent evaluation without exposing answer keys that could enter training corpora. Reproducible evaluations should record the factors that can materially change agent performance: - Model version and reasoning configuration - Prompt and agent orchestration - Sandbox image, packages, and hardware limits - Token, execution-time, and retry budgets - Single-run accuracy and repeated-run consistency The leaderboard is nearing saturation Scores clustered between 75% and 83.1% give CritPt v2 limited room to distinguish the strongest systems. Artificial Analysis plans to include the benchmark in its next Intelligence Index refresh, but future frontier evaluations will need harder problems, independent audits, and published reliability measurements. CritPt v2 supports a specific capability claim: leading models equipped with an offline code sandbox can solve most of the benchmark’s revised, machine-graded physics tasks. It also shows how prompt defects and tool policies can reshape benchmark results. A successor will need to preserve private answers and exact grading while restoring enough difficulty to separate frontier systems.
document fetched 2026-10-23 16:26