Gemini 4 Neon jumps 3x on SWE-sweep (12.8% to 39.0%) solely by changing the harness—are we benchmaxxing the wrong layer?
retained text
Google DeepMind dropped Gemini 4 Neon right into Anthropic's $1.5T IPO news cycle—their fourth periodic-table release in two months after Hydrogen, Argon, and Carbon. Headline numbers look impressive on paper: $3 per 1M input tokens ($0.15 cached), $15 per 1M output, 1M context window, 127 tok/s observed on OpenRouter (roughly 2.76x faster than Carbon's 46 tok/s), #3 on the Artificial Analysis Intelligence Index at xhigh, and #1 on the Coding Agent Index. Two details in the release deserve much more scrutiny than the leaderboard placement. First, the internal pricing inversion: because Google extended promotional $2/$10 pricing on heavier Carbon and Argon through late November, the mid-tier Neon temporarily costs more per raw token than its larger siblings, even though Neon reduces total cost per task by 32% at default high thinking (and 50% to 80% versus rival frontier models on Vals Index and Agent Arena). Second, and far more important for anyone building production coding agents: buried in Google's announcement is the SWE-sweep comparison. On the minimal mini-SWE harness, Gemini 4 Neon scores 12.8%. Inside Google Antigravity, the exact same checkpoint jumps 3x to 39.0%. Meanwhile, practitioners in the wild are still reporting that despite beating Claude Opus 5.5 on static benchmarks, Neon still stumbles on messy multi-file refactors unless the surrounding loop tightly manages context compaction, tool routing, and file-read budgets. If swapping only the harness accounts for a 3x jump on identical model weights, how much of our recent frontier model hype is actually just harness engineering in disguise? For teams running production coding agents right now, where are you seeing bigger real-world gains—upgrading checkpoints or hardening your tool loop and context compaction?
document fetched 2026-11-20 00:20 · canonical https://reddit.com/r/artificial/comments/1y0btfc/gemini_4_neon_jumps_3x_on_swesweep_128_to_390