loading
loading
Which LLM should review your code? New frontier models ship almost every month. Each release can change the balance of quality, cost, and speed available to teams building with AI. Code review brings those trade-offs into focus: how many bugs can a model catch, how much noise does it create, and what does each review cost? At Qodo, review and governance are the product, so model selection is part of our core engineering work as we build our review and governance platform. We run every major model through the same proprietary benchmark, built from real bugs, and use the results to decide what runs in production. This post shares the latest round: 13 models from across OpenAI, Anthropic, Google, and five open-weight families, tested across 38 model and reasoning-level configurations. How the benchmark works Most code review benchmarks rely on synthetic bugs or hand-labeled issues. Ours starts from bugs that actually happened. We take bugs that were later found and fixed, trace each one back to the pull request that introduced it, and ask the reviewer to catch it in that original PR. The current dataset covers 125 pull requests containing 703 known bugs. To isolate the model, we hold everything else fixed. Every configuration runs with a single reviewer, standard review effort, and the explorer subagent turned off. The only variables are the model and its reasoning level. Findings are scored pairwise against the known bugs by an LLM judge (Claude Sonnet 4.6). For each configuration we measure: - Precision: the share of findings that are real bugs. Low precision means noise. - Recall: the share of known bugs the reviewer caught. - F1: the balance of precision and recall, and our headline quality score. - Backtracked bug recall: recall measured only on the bugs traced back from later fixes. - Cost per PR and median review time (P50). What the results show The top tier is tight Four models sit within 4.7 F1 points of each other: Claude Opus 5.5 (56.8%), Claude Sonnet 5.5 (54.0%), Claude Fable 5.1 (53.5%), and GLM 5.3 (52.1%). Below them, a second group between 46% and 50% includes DeepSeek V4 Pro, Kimi K3, GPT-6 Sol, Gemini 3.8 Flash, and GPT-6 Astra. A ranking from one month can reorder the next, which is why we rerun the benchmark with every major release. More reasoning is not always better Raising the reasoning level usually increases recall and lowers precision. The net effect on F1 depends on the model: - Claude Opus 5.5 scores 56.8% F1 at high and 54.0% at xhigh. At xhigh it catches the most backtracked bugs of any configuration (55.7%), but its precision drops to 54.4%. - GPT-6 Sol scores 48.7% at medium and 48.4% at xhigh. Xhigh has higher recall (49.2% vs. 41.4%), but costs almost three times as much ($0.51 vs. $0.18 per PR) and takes more than twice as long (224s vs. 101s). - Claude Fable 5.1 scores 53.4% at low and 53.5% at high, at a similar cost per PR. - Claude Haiku 4.5 is nearly tied at medium (30.3%) and high (30.4%) The best reasoning level is different for every model, and it has to be measured, not assumed. Cost varies by more than 100x Comparable quality comes at very different prices. GPT-6 Luna at xhigh reaches 41.7% F1 for about two cents per PR. Gemini 3.8 Flash at high reaches 47.0% F1 for $3.63 per PR, with a median review time of almost 14 minutes. Claude Sonnet 5.5 at high reaches 51.7% F1 for $0.31 per PR in 90 seconds, which puts it close to the top tier at a third of the cost of Claude Opus 5.5. The highest-scoring configuration and the most practical configuration can differ. Sonnet at xhigh scores 54.9% F1 at $0.73 per PR, compared with 51.7% at $0.31 at high. Sol’s medium and xhigh settings have similar F1, but xhigh trades precision, cost, and time for higher recall. The choice depends on the team’s priorities. Open-weight models are competitive GLM 5.3 scores 52.1% F1, ahead of every OpenAI and Google configuration we tested, at $0.76 per PR. DeepSeek V4 Pro (49.9%) and Kimi K3 (49.0%) also finish ahead of GPT-6 Astra and Gemini 3.8 Flash. For teams that need to run on open weights for data residency, cost, or control, the quality gap to the frontier is now a few points of F1. Models fail in different ways Two models with similar F1 scores can behave very differently in a pull request. Gemini 3.8 Flash at high has the highest precision among the configurations in the table (71.0%) but catches only 35.1% of known bugs. GPT-6 Sol at xhigh is balanced, at 48.4% precision and 49.3% recall. A team that wants few, high-confidence comments and a team that wants maximum coverage should not run the same configuration. What this means for Qodo customers Continuous benchmarking, shipped to production Benchmarking is not a one-time exercise for us. Every new model and reasoning level goes through the same evaluation, and the best-performing combination of model, reasoning level, and harness configuration is what runs in Qodo. When a new release changes the ranking, production changes with it. Customers get the gains from new models without running their own evaluations. Model agnostic, including open weights Qodo is not tied to one model provider. By default we run a mix of providers, choosing the strongest model for each part of the review. When a customer needs a specific provider or an open-weight model, for compliance, data residency, or an existing contract, Qodo runs on that model instead. The results above show that this flexibility does not have to come at a large cost in quality. A decision framework for your existing LLM stack Many engineering organizations already have approved LLM providers and negotiated contracts. This benchmark works as an intelligence index for code review: it shows how each model and reasoning level performs on real bugs, what it costs, and how long it takes. Teams can see what their current stack delivers for review, and where a different model or reasoning level would catch more bugs or cut cost. Adaptive model selection The best configuration depends on the change being reviewed and on what the team cares about: coverage, precision, cost, or speed. Qodo can make that decision on the fly, selecting the best model and reasoning level for each review, so teams get the right trade-off without having to tune it themselves. Methodology notes - Dataset: 125 internal pull requests containing 703 known bugs, traced back from later fixes to the PRs that introduced them. - Configuration: single reviewer, standard review effort, explorer subagent off. Reasoning levels vary by model as listed. - Scoring: pairwise LLM judge (Claude Sonnet 4.6) against the known bugs. - Cost: mean review cost per PR, excluding evaluator cost and failed attempts. - Results reflect runs completed between September and October 2026. Some configurations were run once; others are averaged across repeated passes.
document fetched 2026-10-24 10:06 · canonical https://qodo.ai/blog/which-llm-should-review-your-code