Cognition's FrontierCode 2.0 Reveals Every AI Coder Fails Half of Real Engineering Tasks

retained text

Cognition's FrontierCode 2.0 Reveals Every AI Coder Fails Half of Real Engineering Tasks
Cognition's new coding benchmark tests whether agents respect unstated constraints, with top model Opus 5.5 scoring just 45% on tasks maintainers built over 40+ hours each.
- Cognition launched FrontierCode 2.0, a coding benchmark testing judgment under underspecified prompts
- Top model Opus 5.5 scores just 45%; benchmark is far from saturated
- GPT-6.1 Sol leads cost efficiency at $1.03 per task with 39% score
- Chinese OSS models hit 17%, American OSS models like Beam and Inkling just 3%
- Thoroughness blockers tripled; models often miss unstated but obvious constraints
- Higher reasoning effort can hurt scores by causing scope creep and breaking changes
Cognition, the company behind Devin, has released FrontierCode 2.0, a coding benchmark that evaluates pull requests produced from deliberately underspecified tasks. Its grading covers the work maintainers expect beyond a prompt’s literal requirements, including repository conventions, adjacent edge cases, cleanup behavior, tests, and API stability.
Opus 5.5 leads with a score of 45%, leaving every tested model below half of the available score. The ranking also diverges from older coding evaluations: Muse Wonder 1.0 and Gemini 4 Argon perform strongly on DeepSWE v1.1 but land in the middle of the FrontierCode field.
Judgment enters the rubric
Many coding benchmarks give agents detailed specifications and measure whether the resulting code satisfies them. FrontierCode supplies informal issue descriptions and grades the broader engineering decisions required to produce a mergeable patch. Correctness remains central, alongside design quality and thoroughness.
An agent working from these prompts must infer constraints that an experienced contributor would derive from the repository, including:
- Reusing existing abstractions instead of creating redundant ones
- Following local naming, architecture, and testing conventions
- Fixing related instances of the same defect
- Supporting input types already accepted by neighboring functions
- Preserving public APIs and resource-cleanup guarantees
- Writing tests that fail when the underlying bug returns
Ambiguity reshuffles the leaderboard
Cognition groups models by cost and capability on its live leaderboard. The reported FrontierCode 2.0 results include:
Within these results, even the lowest-cost proprietary models exceed 20%, while the listed open-source groups score 17% and 3%. That separation is wider than the rankings commonly seen on SWE-Bench-style evaluations built around more explicit issue specifications.
A harder rubric and a wider patch
FrontierCode 2.0 expands the repository set, adds categories such as security patches and performance optimization, and applies more extensive correctness and design requirements. Compared with version 1.1, the underlying codebases are 25% larger, while reference solutions modify 67% more files and 89% more lines on average.
Blocking requirements tied to thoroughness appear three times as often as they did in version 1.1. These checks cover obligations such as fixing every occurrence of a defect when the prompt identifies only one, or extending new functionality to input types already handled by a sibling function.
One abandoned generator exposes the gap
One task comes from Mock Service Worker, a JavaScript library for mocking network requests. The issue reports that resetHandlers() fails to restart generator-based resolvers.
Opus 5.5 restores the restart behavior by discarding the old generator reference, allowing the next request to create a fresh generator. That implementation leaves the abandoned generator open, so cleanup placed in a finally block never runs.
function* () {
  try {
    yield 'first'
    return 'rest'
  } finally {
    cleanup() // skipped if the generator is abandoned
  }
}
JavaScript does not automatically call return() when code drops a generator reference. A complete reset must explicitly close the previous iterator, or perform equivalent cleanup, before replacing it. The submitted patch restores the named behavior while violating the resolver’s lifecycle contract.
Higher effort can widen the patch
Cognition also reports that higher reasoning settings sometimes reduce scores because models write more code and modify more files. Larger patches create additional opportunities for unrelated changes, compatibility breaks, and unnecessary redesigns.
A task involving the PostHog Android SDK illustrates the risk. At a low reasoning setting, GPT-6 Astra produces a small fix for a race condition. At the maximum setting, it deletes a public API class, removing the race while introducing a breaking change for downstream applications.
Choosing an agent from these results
Teams evaluating agents for autonomous pull-request work can use FrontierCode alongside benchmarks focused on explicit specifications. The results support several practical decisions:
- Benchmark the intended workflow: FrontierCode is most relevant when agents receive informal issues and operate with limited clarification.
- Compare cost with task quality: GPT-6.1 Sol records a 39% score at $1.03 per task, giving it the strongest reported cost-efficiency result.
- Tune reasoning settings: Higher effort can expand patch scope, so teams should test multiple settings against their own repositories.
- Constrain compatibility risks: Agent instructions and review checks should explicitly protect public APIs, migrations, and cleanup behavior.
- Validate open-source deployments locally: The listed open models trail the proprietary systems on this benchmark and require repository-specific testing before autonomous use.
Maintainer-built tasks add repository context
Cognition says each task was created by a maintainer of the target repository and required more than 40 hours of work. In 80% of the tasks, the benchmark’s reference solution differs from the original upstream pull request. The auditing process also uncovered a previously unreported security vulnerability that was sent to the affected maintainers.
Because Cognition produces both the benchmark and its leaderboard, internal trials remain necessary for purchasing and deployment decisions. Teams should reproduce representative tasks from their own codebases, measure review burden and regression rates, and treat FrontierCode as one signal within that evaluation.

document fetched 2026-10-28 02:52