Opus 5.5 came out this week, so I ran a small experiment against Opus 5. I gave both models the same three normal developer tasks:
retained text
1. Review some code and find problems 2. Write a commit message 3. Write documentation for some code I ran each task several times in two ways: \- with detailed instructions for how to do the task \- without those instructions The answers were scored out of 100 against the same checklist each time. The grader was Opus 5, one of the two models being compared, so keep that in mind. With the instructions, Opus 5 and 5.5 were basically in the same ballpark: \- Code review: 90 vs 91 \- Commit message: 86 vs 84 \- Documentation: 80 vs 66, although 5.5 ranged from 48 to 85 between runs, so I don't think there's enough evidence to say it actually got worse Without the instructions, things got much stranger: \- Code review: 86 → 68 \- Commit message: 77 → 30 \- Documentation: 57 → 57 The 30/100 result made me think 5.5 had completely messed up the commit message, so I went back and read the actual answers. It hadn't. All three were perfectly understandable. They just left out one convention the test expected: starting the commit message with something like \`fix:\`. The checklist caps the score at 30 if that convention is missing. Give 5.5 the written instructions explaining the convention and the same task scores 84 instead of 30. So I definitely don't think this shows "Opus 5.5 is worse." My tentative takeaway is that 5.5 may be perfectly capable of the task, but explicit instructions can still make a huge difference when you care about a particular convention or workflow. This is a small experiment, so I wouldn't generalize too much from it. It was only one example per task, each model ran it 3 to 10 times, and Opus 5 was the grader. But the gap without instructions was large enough that I thought it was worth sharing. Has anyone else noticed 5.5 being more likely to skip specific conventions unless you spell them out? Full results, including every score: [https://driftproofhq.com/reports/011/](https://driftproofhq.com/reports/011/) Disclosure: I built the open-source testing tool I used for this, Driftproof.
document fetched 2026-09-24 00:23 · canonical https://reddit.com/r/ClaudeAI/comments/1wodzzd/opus_55_came_out_this_week_so_i_ran_a_small