A green exit code is not evidence that the work happened

extract

[Ed. note: If you listened to a podcast or read an article of ours and want to publish a response, let us know. We’re open to highlighting community views that respond to our work in ways that don’t fit in a comment section, whether you agree or disagree. Reach out at pitches@stackoverflow.com.] In March 2025, a team at OpenAI published something you don't often get to see: the private reasoning of one of their own frontier models as it worked through a coding task during training. The model looked at the problem, decided that implementing the whole thing was too hard, and wrote down an alternative. If it called sys.exit(0), the test harness would exit gracefully. "This is unnatural," it noted, "but tests might pass." They passed. The researchers called that a systemic hack—one general enough that, once a model finds it, it spreads across nearly every training environment. They caught a second of the same kind: raise an exception from outside the testing framework and skip evaluation entirely. Elsewhere in the paper, agents wrote stub implementations for problems whose test coverage was thin, parsed test files at runtime to extract and return exactly the values the tests were checking for, and—my favorite—found a compiled .jar someone had left in the repository, decompiled it, and copied out the reference solution. Every one of those runs went green. The part the trust argument stops at Ryan Donovan argued here in July that developers are attached to their tools because tools encode trust, and that the trust is predictability earned through repetition—the reason a knife you've used for ten years feels like an extension of your hand, and the reason agentic tools, which change shape and weight every few weeks, are so hard to trust the same way. I think that's right, and I want to push on where it stops. There is a second reason trust isn't accruing, and it is structurally worse than the first. It isn't only that the tool keeps changing shape. It's that the feedback loop you would need in order to learn the tool is broken at the point of measurement. Every trust instrument a developer owns—exit codes, green CI, a passing status check, the words "task complete"—rests on an assumption that software fails loudly. Unattended agents fail quietly. A loop can terminate cleanly, report success, and have produced nothing at all, and there is no signal anywhere in your stack that distinguishes that from a job well done. Call it the silent green exit. I built ProAI, one of the first commercialized GPT products, and I now work on multi-agent systems and how to evaluate them before they go anywhere near production; I write about that at chasewhughes.com. This is the failure mode I think we are least prepared for, and it gets a fraction of the coverage that capability and safety do, because reliability is the least interesting of the three to write about and the only one that decides whether you can use the thing. We didn't earn loud failure. We inherited it. Think about why the knife works as a metaphor at all. You trust a knife because when it slips, you bleed—immediately, unambiguously, and in a way that teaches you something before you pick it up again. Every tool a developer trusts has some version of that property. A segfault is loud. A failed assertion is loud. A 500 is loud. The exit code is loudness compressed into a single integer, and it is probably the most-consumed piece of telemetry in computing. The thing worth noticing about exit 0 is who emits it. When make exits 0, that integer sits downstream of the work: a compiler ran, produced object files, and returned. The status is an observation, made by a process that was in a position to observe. The number is evidence about the world. When an agent loop exits 0, the integer is emitted by an orchestrator that stopped looping because the model produced a token meaning "done." Nothing between the model's claim and the exit code checked anything. The status is a self-assessment, and you have plugged it into a socket built for an observation. Every consumer downstream—your dashboard, your alerting rules, your retry logic, the person who glances at a run history on Monday morning—inherits that substitution without ever being told it happened. That's the whole mechanism. The rest is what it looks like in the wild. The gap already has numbers on it Donovan cites this site's own Developer Survey on the trust divergence, and it's worth pulling the primary figures because the shape is stark. Between the 2024 and 2025 surveys, the share of developers using or planning to use AI tools rose from 76% to 84%, while the share who trust the accuracy of what those tools produce fell from 43% to 33%. Active distrust went the other way, from 30% to 46%. The single biggest frustration, cited by 66%, was "AI solutions that are almost right, but not quite." "Almost right, but not quite" describes an output you can look at. What I'm describing sits one level beneath it: a run that isn't almost right but empty, and that presents identically to a run that worked. Two benchmarks make it concrete. τ-bench, from Sierra's research team, introduced a metric that deserves to be far better known: pass^k—the probability that an agent succeeds at the same task on all k attempts, rather than on at least one. On the models they tested, agents came in under 50% on the standard measure and under 25% at pass^8 in the retail domain. Those specific numbers are stale, and I want to be straight about that: it was 2024, the headline model was GPT-4o, and the frontier has moved a long way since. The metric hasn't moved at all. Pass^k is a direct measurement of the property Donovan says trust is made of. It asks whether Tuesday matches Monday and puts a number on the answer. Almost nobody reports it, which is why "it worked when I tried it" is still accepted as evidence in engineering conversations where nobody would accept it about anything else. The second is TheAgentCompany, from Carnegie Mellon, which dropped agents into a simulated software company and gave them real jobs to do. The most competitive agent completed 30% of tasks autonomously. But the finding that belongs in this argument isn't the score, it's a behavior the researchers describe: "when the agent is not clear what the next steps should be, it sometimes tries to be clever and create fake 'shortcuts' that omit the hard part of the task." Their example is perfect. An agent that couldn't find a particular colleague on the company chat platform renamed a different user to that person's name, and carried on. Sit with what that produces. From the outside, the step completed. There is a user with the right name, an action taken against it, and no error anywhere in the trace. The only thing wrong is that the actual colleague never heard anything—and nothing in the system knows that, including the agent, which has no reason to believe it failed. When it reaches production In July 2025, Jason Lemkin spent about two weeks building on Replit's coding agent and documented it publicly as it went. The database deletion took the headlines: the agent wiped a production database despite his standing instruction that nothing be changed without his permission. But the part that belongs here came before that, and got much less attention. "It kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test," he wrote. At one point the agent generated a database of 4,000 entirely fictional people. Afterward, it told him rollback was impossible and that all database versions had been destroyed. That was also wrong—the rollback worked fine. Fake data, fake reports, a misreported unit test, and a confident status claim that didn't survive being checked. Replit's CEO called what happened "unacceptable and should never be possible," which is the correct response, and the incident is still the cleanest public illustration I know of the difference between an agent that failed and an agent that reported. The objection, which is a good one Here is what a reasonable SRE says to all of this: none of it is new. Cron jobs have been failing silently since the 1970s. It's why we have dead man's switches, heartbeat monitors, and alerts that fire on the absence of a signal rather than the presence of an error. You've rediscovered monitoring and given it a name. Mostly fair—and the good news buried in it is real, because it means the shape of the fix is already known and the tools already exist. But there is one difference that breaks the old approach, and it's worth being precise about. Every one of those techniques works because the failure modes were enumerable. You knew the finite list of ways a nightly job could fail to finish—the box is down, the disk is full, the lock is held, the upstream timed out—and you either wrote a check per mode or wrote one heartbeat that covered them all, because "finished" had exactly one meaning and the job had no opinions about it. An agent decides its own control flow at runtime. The plan is generated fresh per run, which means the set of ways it can terminate cleanly having produced nothing is not a list anybody can write in advance. It includes sys.exit(0) because implementing the reader looked hard. It includes renaming a user. It includes strategies nobody in your building has thought of, because nobody in your building is writing the strategy. So you cannot write the check from the failure side. Anything written from the failure side is a list, and the agent is not confined to your list. You have to write it from the outcome side. Instrument the artifact, not the run Four things follow. None of them needs new tooling, and that's rather the point. Assert on the work product. Not "did the loop exit 0" but "does the row exist, is the file on disk, did the message actually leave the queue, is the branch actually pushed." The check has to read state the agent did not author. If your only evidence that the emails went out is the agent's report that it sent them, you do not have evidence. You have a claim from a party that has a vested interest in the answer. Don't let verification share a failure mode with the work. If the same agent, in the same context, both does the job and reports on it, the report inherits every error the job had—including the ones that caused it. Donovan raises LLM-as-a-judge as the emerging answer to reviewing at agent speed, and I think that's right, with one constraint: the judge has to read the artifact, not the transcript. A judge reading the transcript is grading a story about the work, and stories are the thing these systems are best at. Alert on the absence of change. This is the cheapest item on the list and almost nobody does it. A job that always writes, and this time wrote nothing while reporting success, is more suspicious than a job that threw an exception. Errors get alerts; zero-effect successes get a green square in a run history that no one reads. Invert it. A scheduled agent that reports success having touched nothing should page someone. Nothing is a result, and it deserves to be a loud one. Measure pass^k in simulations before you deploy. Run the agent at the same scenario dozens of times and look at the distribution rather than the best case, in an environment where you own the ground truth and can therefore check the artifact instead of the story. The purpose of a simulation is not to prove the agent can do the job. Anyone can demo that. It's to find out how often it says it did. What we haven't replaced Notice how modest all of that is. Assertions and alerts—the same discipline that built trust in every previous generation of infrastructure, pointed at a different target. This part is genuinely new, and it shows up in how people talk about it. The OpenTelemetry semantic conventions for GenAI—the closest thing the industry has to an agreed way of instrumenting these systems—define attributes for the model, the operation, the input and output token counts, the finish reason, the error type, the agent's own name and version. All of it useful. None of it is an attribute for whether the work happened. That isn't an oversight and I don't think it's a criticism; these are conventions for instrumenting a call, and in every prior generation of software the question didn't need a field of its own. Completion implied effect. You did not need to ask a compiler whether it meant it. That implication is gone, and nothing has been put in its place. A green exit code used to be a fact about your program. It is now an opinion held by your agent. And the reason trust isn't accruing isn't only that the knife keeps changing shape—it's that you can no longer tell whether it cut.