← sources

The agent said it fixed the bug. The original PoC still fired.

source: reddit (reddit:AI_Agents)published 2026-11-03 02:38ingested 2026-11-04 02:36

retained text

A coding agent closes a security ticket. The patch is clean. Tests pass. The diff matches the advisory.

Then someone runs the original PoC against the patched build. It still fires. The agent fixed a different path.

The failure isn't the model's reasoning. It's the check: we verified the diff, not the trigger. The check that would have caught it is simple: freeze the PoC, apply the patch, run the PoC. If the trigger has to change to reproduce, it's a different bug.

Now every rejected action leaves a fixture: the exact trigger, the evidence the agent cited, and the check that would have caught it. The queue clears; the fixture stays.

What do you freeze first when an agent says "fixed"?

document fetched 2026-11-04 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xmh8sp/the_agent_said_it_fixed_the_bug_the_original_poc