← sources

Opus 5.5 writes slop code, but succeeds anyways. A research project inspired by SlopCodeBench

source: reddit (reddit:ClaudeAI)published 2026-10-26 02:05ingested 2026-10-27 02:36

retained text

A few months ago, I ran across this paper proposing a benchmark called SlopCodeBench. The idea is that you give an agent a problem split into a bunch of checkpoints, and at each one it has to add a new feature on top of its own old code without breaking anything. Along the way the benchmark measures how "sloppy" the code gets, using two metrics: structural erosion (how much complexity piles up in a few giant branchy functions) and verbosity (how much of the code is redundant).

The paper found that agent code gets messier the longer the agent works on it, and that agents get worse at keeping everything passing as they go. The obvious takeaway is that the agent is digging its own grave. Messy code is hard to  extend, so it fails.

I wasn't convinced that held for agents, so I tested it. I ran Opus 5.5 through all 196 checkpoints ($266 in API costs, ouch) and checked whether messier code predicted worse results.



**Short version:**

\- Its code did get messier, just like the paper says. Erosion and verbosity both went up in 22 of 35 problems.

\- Messiness didn't predict whether it got the next checkpoint right, once you compare checkpoints within the same problem.

\- It almost never broke old stuff. Only about 0.3% of previously passing tests regressed, and messier repos didn't regress any more often.

\- Messy code didn't make the next change more expensive. Code size did, though: every time a codebase doubled, the next checkpoint cost about 19% more.

\- Across the whole leaderboard, models that write cleaner code don't solve more problems. The correlation is basically zero.

Also, the headline 17% strict solve rate undersells Opus pretty badly. On the tests for each new spec, it passed 84-92%.

Big caveats: this is one run of one model on fairly small programs, and the comparisons are observational. A small effect could still be hiding in there. But if it exists, it's nowhere near the penalty these metrics are supposed to be warning us about.

[Full video link here](https://www.youtube.com/watch?v=gPGYUPubLoI)

document fetched 2026-10-27 00:23 · canonical https://reddit.com/r/ClaudeAI/comments/1xfkhum/opus_55_writes_slop_code_but_succeeds_anyways_a