# i hand-scored my own "did the AI actually do it" checker. it was wrong more often than right — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/7be7a6ad3728e978

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-11-19 03:03 ingested 2026-11-20 02:35

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1xzyri1/i_handscored_my_own_did_the_ai_actually_do_it/)

## retained text



```
follow up to my post from last week about building a checker that watches long AI chats and flags when the assistant says it did something but didn't.



after that post i stopped trusting my own numbers and went back and scored the checks by hand. it was humbling.



\*\*what i found\*\*



\- one check was judging the wrong person. it was supposed to track what the assistant promised to do, but it was mostly tracking what \*i\* said i'd do. my homework, basically. so it was flagging stuff that wasn't the AI's fault at all.

\- another one made claims about work it couldn't see. if the assistant did something outside the chat page, like a file, a commit or a terminal command, the check still had an opinion on it. it scored 3 right, 7 wrong. i retired it.

\- a third had a nice looking 10/0 score that i had to withdraw, because it turned out it was counting things, not checking whether the claim was true.

\- my labelling gate (blind, me scoring without seeing the model's answer) failed twice.



\*\*the trap i walked into\*\*



every time i fixed a check, its accuracy history was meaningless, because the history belonged to the old version. so i was fixing constantly and never had a number i could stand behind.



\*\*what i changed\*\*



i froze the two checks that actually decide the verdict. there's now a test that fails if their code changes before they've been scored on 30 real panels at that exact version. right now they're at 0/30, which is honest at least. the shaky checks still run, but only as info and they don't count.



\*\*re last week's comments\*\*



someone suggested running a pass with only the original ask and the final artifact, since anything the agent wrote mid-run is its own framing. that's a really good point. haven't run it yet, it's next on my list.



someone else talked me out of calling old rows "uncheckable". they're recoverable by hand, just slow. fair.



\*\*the actual question\*\*



for anyone running agents on real work: how do you check the checker? do you spot-check by hand, use a second model, or just trust it?



and if you'd like to try mine on your own chats and tell me where it's wrong, DM me. i need people who'll be mean about it.
```



document fetched 2026-11-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xzyri1/i_handscored_my_own_did_the_ai_actually_do_it
