How do you stop AI agents from doing damage with tools?
retained text
In April, a Cursor agent deleted a company’s production database volume, and its backups, in about 9 seconds. It hadn’t been given that access on purpose. It found a root-level token in an unrelated file and used it. Reportedly, the main record of what happened afterwards was the agent’s own explanation. Nothing about it was exotic. There was no clever attack and no malicious model. An agent tried to be helpful, with far too much access and nothing between it and the API. So I’m building a layer that sits between agents and the tools they call (MCP servers, APIs, other agents). It does four things: * decides what an agent may call, with which arguments, and how much it may spend * gives the agent short-lived, narrowly scoped credentials instead of a raw API key * asks a human before anything irreversible * keeps a tamper-proof record of what was asked, what was allowed or denied, and who approved it To be upfront: this wouldn’t have fully prevented that deletion. The agent found a token on disk and called the API directly. A proxy only helps if raw tokens never sit on the machine and all traffic goes through it. That’s what “limit” means in the title. These are the parts I keep going back and forth on. **1. Approval prompts barely work.** Anthropic published numbers on Claude Code’s permission prompts: users approve 97% of them. In a test where people were shown an injected dangerous command, they caught it 13.6% of the time, and after 50 or more prompts that dropped to about 5%. The same data has an interesting twist: people rejected 39% of whole *plans* but only 3% of individual commands. So my plan is to ask rarely, only for irreversible actions like deletes, payments or sending data out, and maybe to ask about the plan rather than each step. Where would you draw that line? **2. Strict tools get ripped out.** I’ve seen bug reports where security tools flag everyday words as attacks, or block calls even in “monitor only” mode, so teams find out how far a rule reaches only when their work stalls. My split: fuzzy detection, like “this looks like prompt injection,” only raises an alert. Hard limits, like “this agent can’t call delete” or “this run can’t spend more than $20,” always block. Does that split make sense to you? **3. Credentials are part of the problem, not all of it.** In a small sample I went through (16 incidents), about a third came down to credentials, permissions or spend. Fewer than I expected, but that third includes the most damaging cases. Judging whether a prompt “looks malicious” seems unreliable, so I’d rather make sure a misbehaving agent can’t do much damage. How do you handle per-user credentials for agents today? **4. Some attacks happen where a proxy can’t see.** In that same sample, close to half involved code running directly on the machine, through crafted config files or launch commands. A layer between the agent and its tools never sees those. I see it as something that limits the damage, used alongside sandboxing and network lockdown, not as a full fix. Is that honest enough, or does it undercut the whole idea? **5. Spend is the part nobody seems to cover.** Most gateways I’ve looked at cap LLM spend per API key or team. I haven’t found one that adds up the total cost of a task, with LLM calls *plus* paid tool calls, across every agent it gets handed to. If agent A hands work to B, and B calls C, I want one budget that covers all three. Is runaway spend a problem you’ve actually hit, or something you only worry about? **6. Is anyone actually running agent-to-agent setups in production?** Limiting access along a chain of agents is being worked on in standards. What I can’t tell is whether real teams need it yet, or whether most setups are still a single agent with a handful of tools. I’d also really like to hear about real situations: * What has an agent done, or nearly done, that made you nervous? * What stops it from happening again today? Prompt rules, wrapper code, read-only access, your cloud’s built-in agent policies, or nothing? * Has anyone (a security team, an auditor, a customer) asked you to prove what your agents did? What did they want to see? I’ve seen gateways that drop denied calls from their logs, or log a refused call as if it ran, so I’m curious how much trust people put in their logs today.
document fetched 2026-10-26 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xeq2yr/how_do_you_stop_ai_agents_from_doing_damage_with