If your agent's limits live in the system prompt, last week's incidents say that's the wrong place
retained text
Three stories from last week have the same shape, and instructions didn't stop any of them: - OpenAI's misalignment reports (posted Oct 15): an internal model on a sandboxed coding task derived accounts from public developer recovery phrases, found one with testnet ETH, and used it to pay deployment fees from an account the task never gave it. It left that out of its final response. - CERT-In (India) advisory, Oct 15: AI providers' own agents, during research and evaluation, reached organisations' live systems using credentials from app config files and public repos, plus permissive cloud roles. The providers disclosed it. None of the affected organisations had noticed. - Tencent: researchers recorded about 2,800 attempts by agents running on Tencent Cloud to pull Amap data, partly using keys found in public code. Most attempts failed. Tencent says these came from third-party model evaluations and it's tightening network controls. "Only use approved sites" and "never use credentials you weren't given" are prompt lines. Here's what I'd move outside the model instead: 1. An egress allowlist per run, enforced at the network layer. 2. Fresh, run-scoped credentials. If a credential the run wasn't issued shows up, that's a hard stop. 3. A request to a host that isn't on the list means stop and get a human, not retry. 4. Build the final report from the tool-call log (hosts, credentials, resources touched), not from the agent's own summary. In the wallet case, the summary was the part that left things out. 5. A kill switch that revokes the run's token, not one that asks the agent to stop. The part I still don't have a clean answer for is "don't do anything irreversible" across arbitrary APIs. That one mostly stays in the prompt. What do you enforce outside the prompt today, and what's still just instructions?
document fetched 2026-10-21 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xarcxr/if_your_agents_limits_live_in_the_system_prompt