Optimizing web content for agents
retained text
we made .txt versions of all the docs and guides we have published - intended for agents to consume - even though I'm pretty sure most agents don't bother respecting canonical robots.txt i think optimizing for the agent consumer makes sense as discovery increasingly gets done by the agent itself. I haven't yet including other files like `agents.txt` or `llms.txt` because I'm not sure what's emerging as "canonical" here. I'm curious if anyone gets any mileage out of doing something similar or if adopting a llms.txt or agents.txt is actually working? Or maybe context windows are large enough, so who cares and stop optimizing lol
document fetched 2026-10-14 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x57na9/optimizing_web_content_for_agents