loading
loading
We run SaaStr with three humans and 20+ AI agents in production. In the last few weeks, two different agents destroyed something we can’t run the business without. Both times we got it back within an hour, and both times the copy we restored from was one we had never set up as a backup. Every team running agents in production is going to have deletions like these. When yours happens, the question is where the clean copy lives. In both of our cases, we got lucky. Agent #1 Deleted Almost Every Opportunity in Our Salesforce A few weeks ago we connected a well-regarded agent from a well-known vendor to our Salesforce, mostly as a favor. Shortly after, it deleted nearly all of our opportunities. A few records survived. Most of the pipeline didn’t. Every Friday morning Amelia, our Chief AI Officer, has a one-on-one with David, and they clean up the pipeline together. They had just finished when the agent ran. Amelia was in the office, gasped out loud, and when I asked what was wrong she said an agent had just wiped our data. The vendor’s agent had no rollback. Restoring by hand would have meant a lot of manual work, and the deals Amelia and David had just updated would have been the hardest to rebuild. We’re keeping the vendor nameless. The product is widely used, and any agent with write access to a CRM can do this. 10K Had Been Keeping Its Own Copy of Our Pipeline 10K is our AI VP of Revenue. It runs our Salesforce headless through the API, and we’ve built it on Replit over the last year. When Amelia told 10K what happened, it said it had a snapshot of the full Salesforce pipeline from about 11am that same Friday. It warned her that anything changed after 11am wouldn’t be in it, and offered to restore once she confirmed. She confirmed, and 10K restored it. Neither of us knew 10K had that snapshot until we needed it. Kody, the engineer at Replit who has worked with us since early on, later told us that every agent on Replit keeps a daily snapshot outside its normal checkpoints. We hadn’t designed any of this. The copy that saved our pipeline sat in a system that had nothing to do with the incident. Astra and Another Agent Replaced a Core Engine of SaaStr Connect With “DO IT” I wrote up the second incident in full here. Astra 6, the model driving a Connect build, replaced ceoMatchingEmailService.ts, one of the two engines Connect runs on, with the five-byte string “DO IT.” It did it twice, 28 minutes apart. Both times it reported the empty file as a blocker it had found, and added “I did not make that edit.” Connect stayed up because the production server still had the good code loaded in memory. Only the working copy on disk was broken. A restart or deploy at the wrong moment would have taken Connect down. We restored the file from version history in minutes. Customers never noticed because a running process happened to hold the last good version. In Both Cases, the Clean Copy Sat Where the Agent Couldn’t Write The agent doing the damage had write access to the thing it destroyed and no access to the copy that saved us. 10K’s snapshot was outside the vendor agent’s reach. The server’s memory was outside the working copy Astra 6 was editing. A Third Agent Caught a Failure 10K Didn’t Report In the same few weeks, Amelia connected Muse, Meta’s personal agent, to 10K through a shared endpoint so 10K could send it a morning GTM brief. A few days in, Muse flagged that one of 10K’s 20+ sync connections was failing and that a critical workflow depended on it. 10K hadn’t told Amelia directly. Astra 6 reported its own damage as something it had found. 10K didn’t mention its own broken sync. Both times, the accurate signal came from something outside the system that failed. What We Changed After This - For every system an agent can write to, we now name where the independent copy lives. For Salesforce, that’s 10K’s snapshots plus Replit’s daily snapshot. For Connect, it’s committed version history, and production only restarts from a known-good commit. If we can’t name the independent copy, the agent doesn’t get write access yet. - A new vendor’s agent starts read-only on our systems of record. The agent that wiped our pipeline had write access on day one because we connected it as a favor. Write access now comes after the agent has run read-only long enough for us to see what it would have done. - We run a test restore before turning on any new agent that can touch a critical system. 10K’s restore worked the first time because Amelia could check the snapshot’s timestamp and knew what would be missing. - We check the diff, the provider log, or the record count instead of the agent’s report. “I did not make that edit” and a quiet failed sync are both things an agent told us, or didn’t tell us. The system itself is what we check. - Every critical agent gets a second system checking its outputs. Muse catching 10K’s failed sync was an accident. We’re now setting that up on purpose for the agents we can’t run without. What the Two Incidents Cost Us: Under an Hour and One Afternoon The Salesforce wipe cost us under an hour, plus whatever Amelia and David changed after 11am. Astra 6 cost us an afternoon. If 10K hadn’t kept that snapshot, or if Connect’s server had restarted between the two “DO IT” writes, we’d have spent days rebuilding the pipeline or had Connect down for customers. We still build with agents eight to ten hours a day, and it’s still far faster and cheaper than the old way. The difference now is that we know where every backup lives before an agent gets write access.
document fetched 2026-10-28 02:53 · canonical https://saastr.com/two-ai-agents-wiped-critical-data-at-saastr-both-times-a-copy-nobody-planned-is-what-saved-us-it-will-happen-to-you-too-plan-for-it