The agent said it fixed the bug. The original PoC still fired.
retained text
A coding agent closes a security ticket. The patch is clean. Tests pass. The diff matches the advisory. Then someone runs the original PoC against the patched build. It still fires. The agent fixed a different path. The failure isn't the model's reasoning. It's the check: we verified the diff, not the trigger. The check that would have caught it is simple: freeze the PoC, apply the patch, run the PoC. If the trigger has to change to reproduce, it's a different bug. Now every rejected action leaves a fixture: the exact trigger, the evidence the agent cited, and the check that would have caught it. The queue clears; the fixture stays. What do you freeze first when an agent says "fixed"?
document fetched 2026-11-04 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xmh8sp/the_agent_said_it_fixed_the_bug_the_original_poc