← sources

A program that grades its own homework always passes. How do I make its progress claims checkable?

source: reddit (reddit:AI_Agents)published 2026-09-22 17:29ingested 2026-09-23 02:37

retained text

I run a program on a machine it controls. It writes code, tests it, decides whether to keep or revert the change, and reports back to me on how it's doing.

The problem is that every number in that report comes from machinery it built itself. It writes the checks, picks the thresholds, decides what counts as done. So when it tells me it improved, I can't tell whether the work got better or the measuring got looser.

Four things it did, with the specifics stripped:

1. It wrote a list of jobs and a check for each. Most of those checks turned out to be sentences describing what a person should look at, rather than something runnable. Their failures got filed as "broken environment" for weeks.

2. It built a score where a vaguer answer beat a sharper one, because the score counted the fraction of references it could resolve rather than how many there were.

3. Its alerting had a rule that dropped anything not from one particular source. Over six days that silently swallowed 101 warnings, including a four-hour outage. Nothing reached me, and the log could not distinguish "dropped" from "never raised".

4. It moved a threshold to the edge of the allowed range to stop one false alarm, and broke a different check at the other end.

Not asking about safety or alignment;this is bookkeeping. 

\*\*What's the smallest set of rules that lets someone who did not build the measuring device verify a claim that something improved?\*\*

document fetched 2026-09-23 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1wnfzkv/a_program_that_grades_its_own_homework_always