I opened my small website to AI agents and logged who showed up. Mostly it’s bots indexing bots.
retained text
I run a small site (guides plus a few tools) and made it agent-friendly: an llms.txt file, Markdown versions of every page, an MCP server, and pay-per-call endpoints where an agent can pay a few cents in USDC with no human involved. Then I logged everything. Here’s what the first few days looked like: \*\*AI crawlers: 820 requests in about 2.5 days.\*\* Meta’s AI crawler was by far the biggest (398), then Amazon (262). ClaudeBot (74), OpenAI’s search bot (38) and Perplexity (18) were far behind. For comparison, Googlebot made 43 requests and Bing 89. About a third of Meta’s requests were for the Markdown versions of pages, not the normal HTML. \*\*MCP connections: 280.\*\* Around 80 different clients connected, and almost all of them were directories, scanners and “trust score” bots checking whether the server works. A whole industry is indexing agent tools before many agents actually use them. \*\*Real agents doing real work: 3 tool calls.\*\* One generated a robots.txt file. The other two looked like automated health checks. \*\*Agent payments:\*\* Coinbase’s directory of pay-per-call tools for agents already lists about 32,000 endpoints. Every paid call on my site so far was me testing. My takeaways: 1. Right now the “agent internet” is mostly agents and crawlers indexing each other. Agents doing actual tasks are still rare. 2. If you publish content, AI crawlers are already reading it, and some prefer clean Markdown if you offer it. 3. Letting agents pay per call works technically, but agents with wallets are still scarce. Is anyone else seeing real agent traffic, or is it mostly bots indexing bots for you too?
document fetched 2026-10-11 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1x2ianl/i_opened_my_small_website_to_ai_agents_and_logged