# 20 AI agents, hundreds of thousands of test executions, and I'm still figuring out how to keep them accountable — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/8c4e162484526167

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-10-09 05:37 ingested 2026-10-10 02:34

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1x1d6qx/20_ai_agents_hundreds_of_thousands_of_test/)

## retained text



```
Figured I'd share an update on my multi-agent project. I've posted about the general architecture before, but things have progressed quite a bit.

I'm currently about halfway through the third major iteration. Running 20 agents with different responsibilities, a central coordinator, task dispatching, auditing, and a separate machine handling some supporting services.

The funny thing is, getting agents to actually do things isn't really my biggest problem anymore. It's proving they did what they claimed.

I've had tasks marked as verified when the application couldn't even launch. Agents reporting success despite failed checks. Retries duplicating work. Changes breaking unrelated systems. You name it.

At some point I realized adding more agents wasn't going to fix any of that.

So the latest iteration has been focused almost entirely on governance, accountability, and evidence.

Over the past couple of weeks, I've built out a hash-chained audit ledger, deterministic policy checks, action contracts, human approvals, evidence verification, and a separate validation process.

The whole thing is still running in shadow mode. It watches and evaluates what the agents are doing but doesn't automatically block them yet. I'm not giving it that authority until the evidence supports it.

Some numbers from the development records:

\- 20 agents in the fleet.

\- 6,111 recorded test runs.

\- 805,299 test case executions.

\- 4,525 runs performed by review agents, roughly 74% of recorded runs.

\- 19 recorded adversarial review rounds, with 648 findings across four packages.

\- 95.4% mutation kill rate from 5,133 scored mutants in that review snapshot.

\- An independent validator's first live run checked 3,267 records with zero findings.

\- A later validation checked 484 evidence artifacts with zero missing or mismatched.

\- 211 findings accounted for out of 211 in the latest manual triage.

To clarify, the cumulative testing numbers cover the broader development environment since late August, not just this version. They include repeated runs, intentional failures, and regression testing. Some of the review statistics are also older snapshots.

I'm not pretending 800k test executions means 800k different things were tested. Lol.

One of the more interesting moments was the first live pilot. Five actual development tickets went through the new system. Four effects were verified automatically. The fifth triggered a finding that turned out to be a problem with the verification logic itself.

So yeah, I now have a system that checks the agents, and another system checking parts of the system that checks the agents.

And even that isn't perfect.

Just recently we discovered some stale regression tests. The existing suite could pass while not actually testing everything we thought it was testing. That was a pretty good reminder that a green checkmark means absolutely nothing if the check itself is wrong.

I've also gotten much stricter about deployments. Written designs, multiple adversarial review rounds, mutation testing, frozen code snapshots, staged deployments, rollback procedures, and manual approval before production changes.

It's probably overkill for a personal project, but after seeing some of the earlier failures, I'd rather deal with the extra work.

For context, I'm not a professional software engineer. A lot of this has been built using AI-assisted development. I've learned a ridiculous amount just trying to understand why something broke, fixing it, and figuring out how to prevent it from happening again.

There's still a good bit left to implement, including stronger enforcement boundaries and a longer observation period. I'm not calling it finished or fully secure.

Eventually, I'd like to open-source some form of it, but that's a separate discussion.

The biggest lesson so far is that making AI agents useful and making AI agents accountable are two very different problems.

I feel to many projects lack on governance. I'm building from the inside out. 
```



document fetched 2026-10-10 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x1d6qx/20_ai_agents_hundreds_of_thousands_of_test
