← sources

Love opus 5.5 but kimi pro 3.1 tops our own bench

source: reddit (reddit:ClaudeCode)published 2026-10-24 00:25ingested 2026-10-24 11:00

retained text

We built a small in-house benchmark for picking "design panel" models: single-shot reasoning seats we use for ambiguous architecture and UX calls, where there's no single correct answer and the real question is whether a model weighs our actual constraints. It has 6 tasks: a domain-knowledge quiz with a fixed answer key; two past design decisions replayed from the context we had *before* we decided, so the "right" answer is what we actually chose; a code review with planted bugs plus decoys that look wrong but aren't; a house-style rewrite; and a "where should this logic live" architecture tradeoff. A blind judge model scores the open-ended answers with model names stripped, and we score speed and cost against a fixed anchor model. Kimi K3.1 Pro (high) came out on top: capability 87.9 vs 86.7 for Opus 5.5 (medium and high were identical) and 84.3 for Fable 5.1, at about $0.03 a task against $0.08 for Opus and $0.12 for Fable. That made it best overall once cost is weighted in.

What surprised us is *where* the models split. All of them aced the factual quiz and the architecture question, so knowing things isn't the differentiator. The gaps showed up in the messy work. In code review every model found all the real bugs, but Fable and Opus also "found" several that weren't there (precision 0.50), while Kimi was tighter (0.67). On the replayed decisions, nobody fully reconstructed what we chose (0.7–0.8), and the more expensive model could swing 0.6 → 0.8 between identical runs, while Kimi and Opus stayed consistent. Higher reasoning effort bought nothing on this kind of work. Caveats: it's 6 tasks with only some repeated, and the judge is itself a frontier model (Opus), so take the ranking as "cheaper models are genuinely competitive for judgment-heavy design review" rather than a definitive leaderboard.

it goes without saying - fable 5.1 is completely out of date bring on 5.5! 

Model                   Capab.  Speed   Cost  Overall  $/task  
Kimi K3.1 Pro (high)     87.9    47.6  100.0    85.7   0.030  
Opus 5.5 (medium)        86.7    48.7   69.7    80.3   0.075  
Opus 5.5 (high)          86.7    44.2   70.8    80.0   0.081  
Fable 5.1 (med, anchor)  84.3    50.0   50.0    75.7   0.122

document fetched 2026-10-25 00:35 · canonical https://reddit.com/r/ClaudeCode/comments/1xdultw/love_opus_55_but_kimi_pro_31_tops_our_own_bench