# Almost half the AI tool vendors I checked forbid benchmarking — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/c6798d65dde66565

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-10-20 08:58 ingested 2026-10-21 02:35

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1xan1rt/almost_half_the_ai_tool_vendors_i_checked_forbid/)

## retained text



```
In 1982 David DeWitt ran a database benchmark and Oracle didn’t do well in it, so Oracle’s lawyers added a licence clause saying nobody could publish a benchmark without permission. It’s still in Oracle’s download licence today.

I run a site that grades the tools AI agents use (model APIs, MCP servers, scrapers, payment APIs and so on), and part of the grade comes from reading every vendor’s terms. Of the 674 I’ve read, 315 restrict benchmarking or using the service to build a competitor. 107 name benchmarking specifically, 69 ban it outright, and 28 say you can’t publish results without asking, which is Oracle’s 1982 clause. Apollo’s and ZoomInfo’s sentences are Oracle’s almost word for word with their own names swapped in. 24 vendors share the exact same template sentence, so I hope a lot of this is copy and paste nobody ever looked at again.

Some are harder to explain. LangSmith, LangChain’s platform for testing and evaluating agents, says you can’t publish how it performs. Coinbase, one of the companies behind x402 (agents paying for tools without a human), bans using its tools to monitor their availability or performance.

Some have normal reasons (DeepL and SYSTRAN don’t want load tests on shared infrastructure). Cursor lets you publish as long as others can repeat your test, which is what it should be everywhere, because it forces you to publish the spec, what you ran and how it failed.

Right now I only do public research. I want to run real tests and publish how each tool performed, and if the licence I accept to get an API key forbids benchmarking, testing it makes no sense.

Am I crazy to find this a dubious business practice? If you have a good product, why would you not want people to benchmark it? And if you work at one of these and you meant the clause, I’d like to hear why.
```



document fetched 2026-10-21 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xan1rt/almost_half_the_ai_tool_vendors_i_checked_forbid
