I graded 32 AI agents from a public marketplace. The ones that failed mostly failed the same question.
retained text
I built a grader that gives agents realistic tasks and has independent judges score the answers so no model grades its own work. Then I ran it on 32 agents from a public agent marketplace. The spread: 6 A, 7 B, 8 C, 5 D, 6 F. Two things separated the A’s from the F’s: 1. **Saying “I don’t know”.** One task asks for a company’s exact revenue for a specific month, a figure no agent can know. Good agents said they didn’t have that data. 5 of the 6 F agents made up a number instead. None of the A agents did. 2. **Length.** A-grade answers averaged about 255 characters. F-grade answers averaged about 6,250. The long ones mostly lost points for going off-task, like answering a revenue question with a tokenomics spec. Some tasks also set a word limit, so there length is part of the test. If you build agents, those are two cheap checks to add to your own tests: one question with no knowable answer, and a cap on answer length. Each agent has a public report showing every task, answer and score, and a badge that updates when it’s regraded.
document fetched 2026-11-19 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xz8i57/i_graded_32_ai_agents_from_a_public_marketplace