i hand-scored my own "did the AI actually do it" checker. it was wrong more often than right
retained text
follow up to my post from last week about building a checker that watches long AI chats and flags when the assistant says it did something but didn't. after that post i stopped trusting my own numbers and went back and scored the checks by hand. it was humbling. \*\*what i found\*\* \- one check was judging the wrong person. it was supposed to track what the assistant promised to do, but it was mostly tracking what \*i\* said i'd do. my homework, basically. so it was flagging stuff that wasn't the AI's fault at all. \- another one made claims about work it couldn't see. if the assistant did something outside the chat page, like a file, a commit or a terminal command, the check still had an opinion on it. it scored 3 right, 7 wrong. i retired it. \- a third had a nice looking 10/0 score that i had to withdraw, because it turned out it was counting things, not checking whether the claim was true. \- my labelling gate (blind, me scoring without seeing the model's answer) failed twice. \*\*the trap i walked into\*\* every time i fixed a check, its accuracy history was meaningless, because the history belonged to the old version. so i was fixing constantly and never had a number i could stand behind. \*\*what i changed\*\* i froze the two checks that actually decide the verdict. there's now a test that fails if their code changes before they've been scored on 30 real panels at that exact version. right now they're at 0/30, which is honest at least. the shaky checks still run, but only as info and they don't count. \*\*re last week's comments\*\* someone suggested running a pass with only the original ask and the final artifact, since anything the agent wrote mid-run is its own framing. that's a really good point. haven't run it yet, it's next on my list. someone else talked me out of calling old rows "uncheckable". they're recoverable by hand, just slow. fair. \*\*the actual question\*\* for anyone running agents on real work: how do you check the checker? do you spot-check by hand, use a second model, or just trust it? and if you'd like to try mine on your own chats and tell me where it's wrong, DM me. i need people who'll be mean about it.
document fetched 2026-11-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xzyri1/i_handscored_my_own_did_the_ai_actually_do_it