# Optimizing web content for agents — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/5e12df172f481a84

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-10-13 20:21 ingested 2026-10-14 02:36

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1x57na9/optimizing_web_content_for_agents/)

## retained text



```
we made .txt versions of all the docs and guides we have published - intended for agents to consume - even though I'm pretty sure most agents don't bother respecting canonical robots.txt i think optimizing for the agent consumer makes sense as discovery increasingly gets done by the agent itself.

I haven't yet including other files like `agents.txt` or `llms.txt` because I'm not sure what's emerging as "canonical" here.

I'm curious if anyone gets any mileage out of doing something similar or if adopting a llms.txt or agents.txt is actually working?

Or maybe context windows are large enough, so who cares and stop optimizing lol
```



document fetched 2026-10-14 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x57na9/optimizing_web_content_for_agents
