← sources

A successful agent run is not verification. One of our same-model ablations completed 60/60 tasks and got 0/60 correct.

source: reddit (reddit:AI_Agents)published 2026-09-19 13:16ingested 2026-09-20 02:35

retained text

If you’re building agents that modify code, call tools, or execute workflows, there’s a failure mode I think we underestimate:

**An agent can complete its workflow successfully without producing a trustworthy result.**

We hit this pretty hard while running benchmark evaluations:

• **Execution completed:** 60/60  
• **Topology activated correctly:** 60/60  
• **Artifact derived correct answers:** **0/60**

This was a math focused ablation rather than a production coding benchmark, and I’m not claiming the result generalizes to every workflow.

But the architectural takeaway was hard to ignore:

`execution success != correctness`

`self evaluation != independent verification`

Instead of letting the same model decide whether its own patch or output is valid, we’ve been moving toward a stricter separation of responsibilities:

• **Generation:** The agent produces a candidate, patch, or action.  
• **Execution:** The result runs in a controlled environment.  
• **Verification:** External checks inspect builds, tests, static checks, and other configured evidence.  
• **Failure Classification:** Candidate failures stay separate from runner or environment failures when the evidence supports that distinction.  
• **Evidence Receipts:** The system records what ran, the execution context, what passed, what failed, and what remains unverified.

**The model topology is not the trust boundary.**

In a separate provider constrained paired evaluation across 302 units, an adoption gated setup showed positive measured deltas on HotpotQA, MATH 500 / AIME, and MMLU.

But we also saw negative same model cases, which made me increasingly skeptical of unconditional debate or ensemble loops without external verification.

The architecture we’re moving toward looks more like:

`Single agent → External verification → Escalate only when evidence is insufficient`

rather than:

`Agent → Self reflect → Ask itself if it passed → Trust the result`

We’re packaging this reliability and verification layer into **AdaptOrch**

*(Disclosure: I’m the developer.)*

**For people running agent workflows against real repositories or staging environments:**

What is your actual trust boundary today?

Do you rely on prompt based self critique, an LLM judge, isolated execution, deterministic checks, human review, or some combination of them before an agent action is accepted?

document fetched 2026-09-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1wkm0c1/a_successful_agent_run_is_not_verification_one_of