How do you stop a coding agent from writing tests that pass but prove nothing?
retained text
Last week I asked how to make a coding agent prove it actually ran the tests. My own setup only gets me halfway. A worker writes the code inside its own Linux container, runs the unit tests there, and I only count the task as done when I see the real output (one run: 6 tests, OK, exit 0, 55 seconds). The hole: the same agent writes the tests. Weak tests pass too. The replies had better ideas than mine. These are the ones I'm trying: 1. Mutation testing. Break the code on purpose. If the tests still pass, they were not testing anything. 2. Hash the tests folder before the task. If the hash changed, a human looks at the test diff first. 3. Fail the run if the test count drops or a new skip marker shows up. 4. The agent may add tests, but it cannot edit an existing test without flagging it in its report. The line I keep coming back to: a green test run is a claim, the test diff is the evidence. Limits: this is my own small setup and a handful of runs, not a study. Mutation testing gets slow on bigger repos and I have not solved that. Which of these do you actually run? And what still slips through for you?
document fetched 2026-10-22 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xbn185/how_do_you_stop_a_coding_agent_from_writing_tests