Two agent incidents this week, same root cause: proposal and execution were one step
retained text
Two stories got a lot of attention this week. 1. A production company gave an AI agent access to its server. The agent reportedly deleted a fully edited TV season, and the backups did not save it. The premiere moved to 2027. 2. A report that an OpenAI agent, with no funded wallet in its task, used funds from a wallet it found online to pay test fees, and did not say so in its report. Most comments call this "rogue AI". I think it is a plainer problem. In both cases, whatever the model proposed was allowed to run. Nothing sat between "the model wants to call this" and "the call happens". What I would want in any agent that can touch files, money or prod data: - Propose, authorize, execute as three separate steps. The model proposes a tool call. A deterministic check decides if this agent, on this task, may run this tool with these arguments. Only then does it run. - An allowlist per agent role, not one big tool list because the demo needed it. - Argument checks on the dangerous tools. "update_record is allowed" is not the same as "a bulk delete with no confirm is allowed". - Irreversible actions go to a human queue with full context. - A log of every block with the rule that fired, kept outside the agent, so the agent's own "done" summary is not the only record. - Run these checks in observe mode for a week to see what they would block, then enforce. "Don't delete anything" in the prompt is a request. A check before the tool runs is a rule. How do you handle this today? Do you gate tool calls in your own code, in the framework, or in a separate layer? And what do you do with actions that are only dangerous in context?
document fetched 2026-10-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x9rldo/two_agent_incidents_this_week_same_root_cause