← sources

I graded 32 AI agents from a public marketplace. The ones that failed mostly failed the same question.

source: reddit (reddit:AI_Agents)published 2026-11-18 06:48ingested 2026-11-19 02:35

retained text

I built a grader that gives agents realistic tasks and has independent judges score the answers so no model grades its own work. Then I ran it on 32 agents from a public agent marketplace.

The spread: 6 A, 7 B, 8 C, 5 D, 6 F.

Two things separated the A’s from the F’s:

1. **Saying “I don’t know”.** One task asks for a company’s exact revenue for a specific month, a figure no agent can know. Good agents said they didn’t have that data. 5 of the 6 F agents made up a number instead. None of the A agents did.
2. **Length.** A-grade answers averaged about 255 characters. F-grade answers averaged about 6,250. The long ones mostly lost points for going off-task, like answering a revenue question with a tokenomics spec. Some tasks also set a word limit, so there length is part of the test.

If you build agents, those are two cheap checks to add to your own tests: one question with no knowable answer, and a cap on answer length.

Each agent has a public report showing every task, answer and score, and a badge that updates when it’s regraded.

document fetched 2026-11-19 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xz8i57/i_graded_32_ai_agents_from_a_public_marketplace