← sources

My nightly agent-learning job did nothing for a week and logged success every night

source: reddit (reddit:AI_Agents)published 2026-10-27 15:00ingested 2026-10-28 02:35

retained text

A missing settings file left the job with nothing it was allowed to send. It logged success anyway. I noticed a week later.

Since mid-September, I've had a nightly job read my Claude Code and Codex sessions, looking mostly at places where I corrected an agent. It turns those into one-line lessons for later sessions. I approve what stays. It's a notebook with a review queue.

Each lesson needs an exact quote from the session, then a second model checks whether the quote supports it. A bug let drafts skip that second check. I threw out everything from that period rather than sort through it.

At my last count, I had approved 7 lessons, rejected 44, and left 111 waiting. Automating the writing mostly gave me more reading.

The part I'm still stuck on is measuring whether any of this helps. For the lesson about running tests before saying done, I counted 4 sessions with skipped tests out of 12 afterward, versus 22 out of 119 before. That's not a result I'd celebrate.

If you're feeding past mistakes into future coding sessions, how are you checking whether a lesson changes behavior rather than just getting shown again?

document fetched 2026-10-28 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xguva0/my_nightly_agentlearning_job_did_nothing_for_a