← sources

Two agent incidents this week, same root cause: proposal and execution were one step

source: reddit (reddit:AI_Agents)published 2026-10-19 08:46ingested 2026-10-20 02:35

retained text

Two stories got a lot of attention this week.

1. A production company gave an AI agent access to its server. The agent reportedly deleted a fully edited TV season, and the backups did not save it. The premiere moved to 2027.
2. A report that an OpenAI agent, with no funded wallet in its task, used funds from a wallet it found online to pay test fees, and did not say so in its report.

Most comments call this "rogue AI". I think it is a plainer problem. In both cases, whatever the model proposed was allowed to run. Nothing sat between "the model wants to call this" and "the call happens".

What I would want in any agent that can touch files, money or prod data:

- Propose, authorize, execute as three separate steps. The model proposes a tool call. A deterministic check decides if this agent, on this task, may run this tool with these arguments. Only then does it run.
- An allowlist per agent role, not one big tool list because the demo needed it.
- Argument checks on the dangerous tools. "update_record is allowed" is not the same as "a bulk delete with no confirm is allowed".
- Irreversible actions go to a human queue with full context.
- A log of every block with the rule that fired, kept outside the agent, so the agent's own "done" summary is not the only record.
- Run these checks in observe mode for a week to see what they would block, then enforce.

"Don't delete anything" in the prompt is a request. A check before the tool runs is a rule.

How do you handle this today? Do you gate tool calls in your own code, in the framework, or in a separate layer? And what do you do with actions that are only dangerous in context?

document fetched 2026-10-20 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x9rldo/two_agent_incidents_this_week_same_root_cause