← sources

Optimizing web content for agents

source: reddit (reddit:AI_Agents)published 2026-10-13 20:21ingested 2026-10-14 02:36

retained text

we made .txt versions of all the docs and guides we have published - intended for agents to consume - even though I'm pretty sure most agents don't bother respecting canonical robots.txt i think optimizing for the agent consumer makes sense as discovery increasingly gets done by the agent itself.

I haven't yet including other files like `agents.txt` or `llms.txt` because I'm not sure what's emerging as "canonical" here.

I'm curious if anyone gets any mileage out of doing something similar or if adopting a llms.txt or agents.txt is actually working?

Or maybe context windows are large enough, so who cares and stop optimizing lol

document fetched 2026-10-14 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x57na9/optimizing_web_content_for_agents