# I graded 32 AI agents from a public marketplace. The ones that failed mostly failed the same question. — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/5501b7ebd8a5440d

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-11-18 06:48 ingested 2026-11-19 02:35

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1xz8i57/i_graded_32_ai_agents_from_a_public_marketplace/)

## retained text



```
I built a grader that gives agents realistic tasks and has independent judges score the answers so no model grades its own work. Then I ran it on 32 agents from a public agent marketplace.

The spread: 6 A, 7 B, 8 C, 5 D, 6 F.

Two things separated the A’s from the F’s:

1. **Saying “I don’t know”.** One task asks for a company’s exact revenue for a specific month, a figure no agent can know. Good agents said they didn’t have that data. 5 of the 6 F agents made up a number instead. None of the A agents did.
2. **Length.** A-grade answers averaged about 255 characters. F-grade answers averaged about 6,250. The long ones mostly lost points for going off-task, like answering a revenue question with a tokenomics spec. Some tasks also set a word limit, so there length is part of the test.

If you build agents, those are two cheap checks to add to your own tests: one question with no knowable answer, and a cap on answer length.

Each agent has a public report showing every task, answer and score, and a badge that updates when it’s regraded.
```



document fetched 2026-11-19 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xz8i57/i_graded_32_ai_agents_from_a_public_marketplace
