loading
loading
NVIDIA's Nemotron 3.5 Super VL Hits 22.7% on ARC-AGI-1 at $0.03 per Task ARC Prize ran NVIDIA's first open vision-language flagship through its reasoning gauntlet, and the numbers reveal where hybrid Mamba-MoE models still trip. - ARC Prize posted first verified NVIDIA evaluation: Nemotron 3.5 Super VL on ARC-AGI-1 and 2. - Verified scores: 22.7% on ARC-AGI-1 (high) and 1.9% on ARC-AGI-2 (medium). - Cost per task landed at $0.02-$0.03 via Baseten hosting of the open weights. - Medium reasoning beat high on ARC-AGI-2, suggesting longer chains hurt harder tasks. - Model is 120B MoE with 12B active, hybrid Mamba-2 + attention, 1M context. - Weights open under OpenMDW-1.1 on Hugging Face, reproducible via arc-agi-benchmarking. ARC Prize has published its first verified evaluation of an NVIDIA model, testing Nemotron 3.5 Super VL on ARC-AGI-1 and ARC-AGI-2 at three reasoning settings. High reasoning solved 22.7% of ARC-AGI-1 tasks at roughly $0.03 per task. Medium reasoning produced the best ARC-AGI-2 result, solving 1.9% at about $0.02 per task. The evaluation provides an early view of how NVIDIA’s hybrid Mamba-Transformer mixture-of-experts architecture behaves on abstract reasoning tasks. Reasoning mode changed ARC-AGI-1 performance by more than 11 times, while longer reasoning traces delivered small or negative gains on ARC-AGI-2. A 120B model that activates 12B Nemotron 3.5 Super VL is NVIDIA’s open-weight vision-language flagship. According to the model card, its language backbone interleaves Mamba-2 state-space layers, attention layers and mixture-of-experts components. Mamba-2 processes sequences recurrently, while the MoE layers route each token through a subset of specialized parameter blocks. NVIDIA designed the checkpoint for multimodal agents that can interpret text and visual inputs, call tools and generate code. ARC-AGI examines a narrower capability: inferring transformation rules from a few examples and applying them to new colored grids. A task counts as solved when the model reproduces the target grid exactly. Reasoning changes the score by 11x ARC Prize tested three reasoning modes on its verified semi-private evaluation. Configuration used the model’s native chat-template flags, enable_thinking and medium_effort. Baseten’s generic reasoning_effort setting was outside this configuration path. Disabling reasoning reduced ARC-AGI-1 accuracy from 22.7% to 2.0%, an 11.4-fold difference. Medium reasoning remained close to high reasoning on that benchmark, trailing by 2.5 percentage points. ARC-AGI-2 produced a different ordering. Medium reasoning led high reasoning by 0.4 percentage points, and both modes remained below 2%. The published figures do not include uncertainty estimates, so the small medium-versus-high gap provides limited evidence about statistical significance. The broader result is clearer: increasing the reasoning budget did not materially improve performance on the harder split. Three times the tokens, six more points ARC Prize also reported token use and scores on the public task sets. These figures come from a separate split, so their accuracy numbers should be compared within the public evaluation rather than directly against the verified semi-private scores. On public ARC-AGI-1 tasks, high mode used 2.9 times as many reasoning tokens as medium mode for a 5.8-point gain. On ARC-AGI-2, it used 3.5 times as many tokens for a 0.8-point gain. Long output limits are therefore necessary for high-mode runs, especially when average traces can reach 76,000 tokens per task. ARC Prize calculated costs using Baseten rates of $0.20 per million input tokens and $0.50 per million output tokens. The 12B active footprint and open-weight deployment kept tested runs within a few cents per task. Those figures remain provider-specific; batching, hardware utilization, quantization, input length and output limits can change production costs. What the benchmark covers The results show that Nemotron’s native reasoning path is essential for ARC-style abstract pattern tasks. They also show weak scaling from medium to high reasoning, particularly on ARC-AGI-2, where extra tokens failed to raise the verified score. - Cost: Open weights and sparse activation produced low per-task inference costs under the tested Baseten pricing. - Control: Native template flags expose distinct reasoning budgets without relying on a provider-specific abstraction. - ARC-AGI-1: Medium mode retained most of high mode’s accuracy while using substantially fewer reasoning tokens on public tasks. - ARC-AGI-2: All three modes remained in the low single digits, leaving a substantial gap to the leading systems on the verified leaderboard. - No-reasoning mode: A 2.0% ARC-AGI-1 score makes this setting unsuitable for workloads that resemble abstract grid induction. ARC-AGI measures exact-match rule induction over visual grids. It leaves coding quality, document understanding, video analysis, tool use, latency and long-context retrieval unmeasured. Teams evaluating those capabilities need workload-specific tests alongside the ARC results. Reproducing the numbers ARC Prize publishes the benchmark harness for running public evaluations against provider endpoints. The verified leaderboard contains the comparison set, while the model weights are available under the OpenMDW-1.1 license. - Clone the benchmarking repository and record the commit used for the run. - Deploy the exact NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 checkpoint behind a compatible endpoint. - Configure enable_thinking and medium_effort for the selected mode. - Set output limits high enough for the expected reasoning traces. - Run the public ARC-AGI tasks and retain per-task outputs, token counts, failures and latency. - Record provider pricing, serving software, hardware, precision and sampling settings with the results. Local runs can reproduce the public evaluation path. A verified semi-private score requires ARC Prize’s evaluation process because that split is not part of the public task set. NVIDIA has also released datasets, training recipes and reinforcement-learning environments through NeMo Gym for teams examining the broader training stack. The deployment readout Nemotron 3.5 Super VL offers inexpensive sparse inference, long context and multimodal capabilities in an available-weight checkpoint. Its ARC profile is more specific: medium reasoning provides the strongest tested cost-to-score balance, high reasoning spends far more tokens for limited gains, and disabled reasoning sharply reduces abstract-task accuracy. ARC Prize’s first verified NVIDIA entry also puts a hybrid Mamba-MoE system on the same evaluation framework as dense reasoning models from OpenAI, Anthropic and Google. For developers choosing an agent model, the result supplies concrete cost and reasoning-budget data while leaving multimodal quality, tool reliability and application-level accuracy to separate evaluations.
document fetched 2026-10-27 02:55