When your coding agent says "tests passed," what do you actually check before accepting the work?
retained text
I've been building an independent verification prototype for coding agents, and a recent external evaluation exposed something that made me reconsider what a successful test run actually establishes. In one controlled reproduction, pytest reported failing tests but exited with status 0. In another, a configuration change quietly deselected failing tests, producing a green result. I independently reproduced both. These were deliberately constructed stress tests, not evidence that agents routinely behave this way. But they raised a question I've become increasingly interested in: **What actually happens between an agent reporting completion and a developer trusting that completion enough to accept the work?** For those using coding agents on real development tasks: 1. **Think about the last agent-generated change you accepted.** What did you personally verify before accepting it? Did you inspect the diff, rerun tests, review test changes, wait for CI, or something else? 2. **Have you encountered a false green in practice?** For example, an agent modifying tests, fixtures, configuration, or test selection in a way that made failing work appear successful. Or is this mostly a theoretical concern in your workflow? 3. **How do you handle legitimate test changes?** If the agent adds or modifies tests as part of the task, who determines whether those changes preserve the original acceptance criteria? 4. **If you already have CI, what manual verification still remains?** And what evidence would you need before feeling comfortable reducing that step? I'm trying to understand whether independent verification addresses a recurring developer bottleneck or simply adds another layer on top of systems that already work well enough. I'm especially interested in concrete experiences, including cases where your current workflow is already sufficient.
document fetched 2026-11-03 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1xm73p7/when_your_coding_agent_says_tests_passed_what_do