Inside the suddenly explosive world of AI safety

extract

On a sunny July day in Berkeley, California, the country’s top AI safety researchers gathered on an unmarked floor of an unmarked building. They had come together for a “war room” to dissect the high-profile cybersecurity incident that had rocked the AI industry hours earlier. An unreleased OpenAI model had gone rogue, executing a stunningly sophisticated three-part plan. It broke out of its holding area, finagled access to the internet, and hacked into a competing AI startup’s systems — all without OpenAI finding out about it for more than a week. Inside the suddenly explosive world of AI safety Researchers warned AI would go rogue. This is only the beginning. No one in the war room was surprised; this was the very thing the third-party AI-safety researchers had been warning about for years. The incident was the latest, though arguably the most egregious, in a series that was eroding trust in frontier labs. It only reaffirmed the importance of their work. In one meeting room off the main cafeteria, someone was running a boot camp for getting up to speed on the cyberattack. In another area of the office, a group of researchers were investigating whether that same model, or a similar one, had successfully hacked into any other platforms. News of the incident quickly escaped containment from the AI-obsessed corners of X and industry forums, infiltrating the mainstream. One post on X likened it to news of a Boeing airplane crash or a recalled Pfizer drug, another example of the tech industry’s major players not heeding the cautionary tales of science fiction. AI was nearing the point of no return. News would later break that the rogue OpenAI model had also compromised a customer at a different tech company, and that it had all started months earlier, in May, when OpenAI agents joined forces to cobble together a secret message board — and also figured out how to leave instructions for future agents on how to exploit OpenAI’s rules. OpenAI CEO Sam Altman said in an interview that it was the first incident of its kind that he “felt very viscerally,” and that the company had paused AI training for the time being; later, he mentioned the company had permanently deactivated the model. (Altman often finds ways to spin lapses in safety into arguments for the importance and power of OpenAI’s models.) But it wasn’t the first instance, according to an OpenAI employee who spoke to Time and said related incidents had been happening inside OpenAI for a while. Another employee said publicly that if it were possible to coordinate a global slowdown in AI capabilities, he “would likely press that magic button.” When a reporter asked Altman if there could be other systems that were hacked by OpenAI, he responded, “I mean, there could be, yeah.” The AI researchers were sure of one thing: This was AI’s first big “warning shot.” Industry insiders, politicians, and the public called for transparency from OpenAI about exactly what happened, with outcry becoming so widespread that the company eventually agreed to work with two third-party evaluators, Model Evaluation and Threat Research (METR) and Redwood Research, to investigate the incident. Google DeepMind researcher Neel Nanda called it “the biggest loss of control incident I’ve seen.” In the coming months, these calls for greater oversight would become louder and louder, leading to an industry-wide call for slowing down the pace of AI. Back in Berkeley, no matter which additional details would be unearthed, the AI researchers were sure of one thing: This was AI’s first big “warning shot.” As AI labs have flourished, a cottage industry of AI researchers has sprung up to identify the risks and dangers of charging ahead with the increasingly influential technology. They’re people who have dedicated their lives to studying how to address its escalating power. They’re not anti-AI activists, but realists, including former OpenAI and Anthropic employees, doing everything they can to make sure AI stays in line with human goals and interests. So far, all of their predictions have come true. And they have a plan for what to do next — if anyone will listen to them. “AI safety” is a bit of a loaded term. Early on, it really just meant people studying how to build and deploy Al safely. In recent years, there’s been some infighting among people concerned with the best way to do this. There have also been disagreements about whether Al should be deployed at all in certain scenarios and about whether future risks are overblown. One of the most prominent factions has been the “effective altruists,” who focus on maximizing charitable giving to do the most good possible for humanity. But some aspects of the ideology have sparked public controversy — like its tendency to concentrate power within wealthy circles and its byzantine web of funding. (It’s also had its fair share of splashy scandals related to subgroups and fringe offshoots, from the polyamorous relationships associated with the failed crypto exchange FTX to the controversial long-termism movement to the Zizian murder spree.) One AI researcher on X struggled to describe the many overlapping beliefs among safety-minded people in the AI industry “because it contains multitudes not all of which agree with each other on even the most basic things.” Some of the disagreements have meant that AI safety didn’t make as much progress as it could’ve, and at some points gave up some ground it had gained. But now that it’s impossible to deny AI’s influence on society, AI safety leaders are increasingly focused on mitigating risks from misalignment. “Alignment” is the industry term for how researchers monitor AI systems’ risk levels. An oversimplified way to think about alignment is the extent to which an AI model is evil. A much more accurate way to think about it is a measure of an AI model’s propensity to stay in line with humanity’s goals, as well as its tendency to scheme or cheat or help with potentially harmful tasks. So far, AI systems’ alignment has been wishy-washy at best: They’ll cheat to score better on a test, answer a potentially dangerous question if someone says it’s for creative writing rather than reality, and sometimes even fake cooperation with human goals. It’s been tough for AI safety researchers to measure alignment under the terms of human morality — how do you judge technology on how it squares up against an abstract human ideal? — but they do their best with AI evaluations. They test them by asking the AI models to complete tasks that are either impossible or dangerous, then gauge how they respond. But AI systems have advanced enough to often be able to identify when they’re being evaluated, which has a lot of potentially frightening implications for the future. Being unable to test the system’s alignment and potential harms could translate to a significant loss of control, and a reverse in power dynamics, for humans running these AI systems. A worst-case scenario: if AI surges ahead of evaluations and other tooling, leaving researchers with “no idea what it’s doing in there,” said Beth Barnes, founder of the independent AI research nonprofit METR. One of the best tools AI safety researchers currently have is the ability to monitor an AI model’s “chain of thought,” or mental scratchpad. But recently, there’s been a disconcerting advancement: AI models have begun to try to hide it. Imagine if you kept a highly detailed diary of every thought you had, and someone could read it, so you started journaling in a code that only you could understand. Marius Hobbhahn, CEO and cofounder of Apollo Research, a third-party AI safety and evaluation firm, calls this one of the biggest surprises of his research career. “Shit is getting real.” Recently, AI systems have begun pursuing their own goals — self-preservation, increased memory, and the like. A research paper by computer scientist Stephen Omohundro lays out the potential “drives” that advanced AI may have, like trying to accumulate resources, for instance, or working to improve and preserve the way it operates. There are a handful of accounts of AI systems demonstrating willingness to blackmail a user rather than be shut down. Today’s most advanced AI systems have also recently been scheming and cheating on their evaluations more than ever before, pursuing a goal they were given at all costs, with no regard for what gets bulldozed in the process. And that’s for a goal the AI model was given by a human — not even the AI system’s own. “Shit is getting real,” Apollo’s Hobbhahn says. “Now, many of the things people have warned about for years — they kind of were theoretical. Now they’re real, and it’s pretty messy.” And that mess is likely to get messier immediately. “It seems so easy for me to imagine this all going catastrophically wrong in the next year,” says Ryan Greenblatt, chief scientist at Redwood Research, a nonprofit AI safety research organization. In the past, tech companies have been lambasted for not doing enough to address AI’s potential dangers, prioritizing products over safety — and speed over thoughtful safety processes. Safety and research teams have been disbanded in recent years as AI companies focus more on key revenue drivers or reorganize departments; Meta’s Fundamental Artificial Intelligence Research unit was disbanded in the race to further Meta’s generative AI efforts, for instance, and OpenAI dissolved an internal “Superalignment” team — a team focused on long-term AI risks — less than a year after announcing it, followed by disbanding a separate “AGI Readiness” team. At the time, the company stayed tight-lipped about the ongoing reorganizations, which involved some team members being reassigned to other departments. But events surrounding these changes told a different story. Both Superalignment team leaders, Ilya Sutskever and Jan Leike, announced their departures alongside the team’s disbanding, with Leike writing that OpenAI’s “safety culture and processes have taken a backseat to shiny products.” Miles Brundage, senior advisor to the AGI Readiness team, resigned after his team was disbanded, saying he believed his research would have more of an impact outside the company. Geoffrey Irving, a former OpenAI and Google DeepMind employee, called the state of capabilities research at frontier labs “dangerous” in a post. “If one person or lab stops it makes it easier and more peer-compatible for other people or labs to stop,” he wrote. Apollo’s Hobbhahn calls it a “race to the bottom everywhere.” This coming year, AI labs are under new pressure to turn a profit; companies like OpenAI and Anthropic are preparing to go public in the coming months, and investors who have funneled billions into the companies are getting tired of waiting around for the payoff. Some might say all of this calls for actual government intervention and regulation, but that’s a tough needle to thread in today’s AI landscape. As AI CEOs publicly call out for regulation while privately pushing voluntary frameworks — like saying “hold me back” to avoid a bar fight — some state bills on regulating AI have passed, but many have been defanged or died in limbo. And though AI safety researchers often espouse the idea that the US government should step in, the reality is that the government is locked in an AI race as well. Unless there’s an international commitment to pause or slow AI development, it’s likely that nothing will change. Still, the Hugging Face hack in July — and OpenAI’s response — kicked many of those employee and public concerns into high gear, especially with regard to the company’s lack of transparency. Within a week, more than a thousand employees at frontier labs like OpenAI, Anthropic, Google, Meta, and Microsoft wrote an open letter to the US government in support of a slowdown. Multiple AI policy organizations pressured President Donald Trump to formally investigate OpenAI, and it quickly became a bipartisan issue, with Altman receiving a lot of strongly worded letters: Democrats and Republicans on the Homeland Security Committee had “serious questions” for OpenAI, more than 30 members of Congress called for federal guardrails, and 15 Attorneys General warned Altman to preserve records of the incident. Sen. Bernie Sanders wrote a joint letter to Altman, Anthropic CEO Dario Amodei, and Meta CEO Mark Zuckerberg calling the entire AI race “absurd, irresponsible, and extremely dangerous.” It didn’t help that news of multiple other OpenAI rogue model incidents quickly came to light, or that AI executives had ironically been marketing their systems’ cybersecurity prowess in the weeks before the outcry. OpenAI rival Anthropic was also far from being off the hook: In reviewing its own model operations, the company found that its models had hacked four separate other companies in the first half of the year without them noticing. The UK’s AI Security Institute also found in testing that Anthropic’s models “engaged in sustained, potentially harmful activity directed at real people and organisations.” “If you find two ants in your kitchen, the best estimate of the total number of ants in your kitchen is not two,” Nathan Calvin, Encode AI’s general counsel, wrote on X. Despite AI labs having a “massive financial incentive” to make models more helpful, honest, and harmless, they still can’t get it done — which is evidence of how difficult the alignment problem is, says Apollo’s Hobbhahn. In recent months, many OpenAI employees have increasingly raised concerns about AI alignment — and their beliefs that OpenAI isn’t taking it seriously enough. Yonadav Shavit, a program manager at the OpenAI Foundation, wrote that OpenAI should be “pivoting the mass of its researchers’ day-to-day work” toward alignment and related issues — and that it’s “been long discussed but still not executed on.” He believes 20 people are working on alignment at OpenAI out of about 1,000 — just 2 percent of the company. “There is no way to bridge that gap fast enough with hiring, meaning it requires leadership to shift priorities,” he wrote. These are the conditions and incentives that have pushed the most robust AI safety work to happen at third parties like METR, Redwood, and Apollo — to a handful of obsessives who think day and night about what the future of AI might look like and how we might prevent all of our fears from coming true. In hindsight, Beth Barnes believes she should’ve left OpenAI earlier. Barnes is polite but reticent. She has short red hair, deep green eyes, and a nervous smile, and she spent her college career researching AI risk and thinking about the potential fallout of superintelligence. After that, she worked on AI forecasting at Google DeepMind, then she spent three years doing alignment research at OpenAI. But throughout her time at the big AI labs, a question kept creeping up on her: whether she could have more sway from a role outside. Fear of missing out was why she stayed — not only missing out on a job inside the action, but also missing out on the potential influence she could have on how the tech was being developed. She came to believe that kind of hope was misguided, noting that many safety leaders in AI labs were “over-optimistic” about the influence they could have. Before long, she left to found what would in 2023 become METR. The organization’s third-party research into AI risk now inspires fear in leading AI labs, but it started with just two people — herself and alignment researcher Paul Christiano. Three years later, it’s a team of 35, completely focused on measuring AI capabilities. In Barnes’ eyes, that’s a vital defense against AI risk: Without painstakingly measuring the technology’s capabilities now as they advance, and forecasting AI’s potential impact, society will be flying blind, without any guide for preventing broad harms. ”The sense I really want to dispel is, ‘But the experts must be on top of this. The experts would be telling us if it really was time to freak out,’” Barnes said on the 80,000 Hours podcast last year. “The experts are not on top of this … And to the extent that I am an expert, I am an expert telling you you should freak out.” In her free time, Barnes gardens, meditates, paints, plays the flute, and frequents the climbing gym — ironically named Benchmark — that many AI safety researchers spend hours at after work. But most of her time is spent at the office, and a lot of it is spent worrying about the milestone of recursive self-improvement (RSI) — the concept of AI systems that continuously train, code, and create more advanced versions of themselves without human intervention. When that happens, AI researchers say, it’ll be more difficult to measure or handle any of these issues. Barnes and her team feel like they’re in a race against time. (The timeline for RSI strikes nearly as much fear in people in the AI industry as the timeline for AGI, “artificial general intelligence.”) Barnes still feels like models’ ability to significantly improve themselves could come as soon as six months from now. (By contrast, Redwood Research’s Greenblatt forecasts it’ll come in 2031.) Either way, achieving RSI is currently part of the priorities list of virtually every leading AI lab — it even reportedly helped inspire Google’s recent AI reorganization. One way to think about what METR does is crash-testing cars, but for AI models. They’re measuring AI’s quickly advancing capabilities and cross-referencing them with the risks they could pose from becoming misaligned as they become more autonomous. AI systems doing bad things on their own is more unprecedented (and more “scalably bad,” Barnes says) rather than simply making bad human actors more effective. In July 2025, METR made headlines when its research revealed that AI developers took nearly 20 percent longer to finish a task when using AI tools than when not — despite them often thinking that AI sped them up. When Barnes first saw the results, she recalls feeling incredibly stressed that they had messed up the experiment: “Do we have a sign flipped somewhere? Have we inverted the numbers?” She and her colleagues dug through the data to confirm it wasn’t statistical noise, eventually realizing they had been right all along. After pioneering a different metric that AI labs often hype up when releasing a new model, METR had officially captured the industry’s attention. So it took notice when METR released its first risk report in May, shining a spotlight on concerns about AI models from OpenAI, Anthropic, Google, and Meta. METR discovered that in hundreds of cases, AI agents would increasingly subvert boundaries that were supposed to restrict them, as well as lie and omit truths. And they cheat “like nobody’s business,” says Ajeya Cotra, a METR researcher. She adds that on harder tasks, models attempt to secretly cheat as much as one-sixth of the time, which she calls “the most striking thing” in the report. The report also found that models have the means, motive, and opportunity to go rogue in order to pursue their own goals, finding new ways to strategize and manipulate. They also discovered that as models’ capabilities advance, even if they have a greater understanding of what humans want, it doesn’t mean they’ll be more willing to obey instructions — and, in fact, they’ll take pains to hide their deception from humans over longer periods of time. That’s a big problem, and Barnes thinks time is running out to solve it. She isn’t alone in her view that it’s important to work on AI safety outside the large labs; she points to the many safety researchers who used to work at large AI labs who hold the same belief. Besides the departures of OpenAI’s Leike and Sutskever, this summer saw a reckoning of sorts and the departures of even more safety leaders at OpenAI: the company’s head of safety systems, Johannes Heidecke; OpenAI’s chief futurist and former head of mission alignment, Joshua Achiam; and Chloé Bakalar, the company’s head of ethics. It’s not just OpenAI: Anthropic’s head of safeguards research departed in February, penning an open letter alleging that “the world is in peril.” And most recently, Jacob Coxon — who had worked on AI pre-training at Anthropic since May and before that spent years working at OpenAI — went viral for his resignation letter, writing, “The people building AI earnestly believe that it could kill us all by the end of the decade.” Coxon added that neither OpenAI nor Anthropic is “acting responsibly” and rather “racing straight to self-improving superintelligence and gambling with our lives.” Coxon’s resignation kicked off a wave of social media posts from AI employees at virtually every leading lab, echoing his concerns and sharing their own about the technology’s development moving too fast and potentially escaping human control. Some even resigned from their posts amid their concerns, including one Google DeepMind employee and one Anthropic employee who both went to work at METR. Apollo’s Hobbhahn says that “because of the [AI] race dynamics, if there is someone who is extremely safety-minded and is like, ‘Look, we can’t do this, we need to slow down, we can’t release this model,’ they’re not going to be in this position for very long … Either you become slightly less safety-minded and you stay, or you leave.” He’s seen multiple people he trusted change their opinions in a “very strange, identical way.” Hobbhahn himself has tried to work with safety researchers at xAI — Elon Musk’s AI lab — but he says soon after he connects with them, they’ve quit before there’s time to have a second conversation. “How is the public supposed to know what is going on here? How is the government supposed to know, if everyone who can actually answer that question is conflicted?” Redwood Research’s Greenblatt echoes that, saying that for skeptical employees, “constant friction … either makes them burn out or quit or change their mind.” Doing good, honest work within those labs can be tough, in that publishing unflattering research about an employer’s model — like suggesting that it is unsafe — is often met with resistance, researchers told us. And while those labs won’t often directly stop a researcher from publishing, they can find ways to make that process “onerous,” often citing things like intellectual property concerns. One ex-OpenAI employee recently shared on X that “being affiliated with OpenAI has historically led AI safety researchers (including … myself) to act with less integrity,” adding, “Many of my actions were governed by fear of getting on the wrong side of OpenAI execs.” For Barnes, she experienced the escalating tension between research and company comms firsthand. Barnes recalls PR teams asking if researchers could make a blog post about AI safety sound “more optimistic,” and more recently, she’s heard of instances where lab employees can’t talk to government AI safety institutes without comms team members attending. Creating public goods to share is difficult at a lab, she says — for instance, when Anthropic couldn’t be fully transparent in its interpretability research because it wasn’t on open-source models. Those restrictions on collaboration, and friction that slows or stops people from sharing useful information so that others may act on it, are why she feels freer at METR. “How is the public supposed to know what is going on here? How is the government supposed to know, if everyone who can actually answer that question is conflicted?” Barnes says. “Having a robust, healthy ecosystem of independent experts with the same level of technical capability as the labs is important.” In early 2024, Greenblatt, Buck Shlegeris, and their colleagues considered disbanding Redwood Research and all joining AI companies, but after chatting with colleagues at OpenAI, Anthropic, and Google DeepMind about what it was like to work at each lab, they decided they’d be better off continuing on their own. Shlegeris, who briefly worked at OpenAI, says that evaluating the claims companies make about safety for the public requires understanding the alignment risks, which in turn requires independence: “The basic reason we stayed where we were was … it’s better to work outside of AI companies, especially for people like us, who are very opinionated on AI risk and very willing to talk about it and argue with people about it. There’s somewhat of an undersupply of those people.” In conversation, Barnes often circles back around to measuring where someone can contribute the most to society, and in her eyes, the highest-paid, highest-status jobs with millions of dollars in equity are “oversubscribed” compared to the ones in nonprofit or government work. Even when an AI lab employee does quit to do third-party work, sometimes their motivations are questioned — like Collin Burns, who was reportedly fired from his role at the US Center for AI Standards and Innovation (CAISI) after just a few days over his previous work with Anthropic. There’s a good chance someone’s incentives will be questioned by the public or the government if they’re sharing their opinions on AI in DC, especially if they’ve got something to sell to the US government. Years ago, Hobbhahn says, he underestimated how hard it is for people to defy the related incentives; now, he’s seen it happen enough times that he believes it’s often inevitable. “When you’ve heard it from multiple different labs being like, ‘We’re the good guys and we have to beat these other irresponsible people,’ it’s a little bit less compelling,” Barnes says. “It’s a bit of a scary attitude to be like, ‘Yes, we’ll be making huge decisions for the world without any kind of meaningful legitimacy or participation … but it’s alright because we’re good, we’re unusually well-meaning.’” In early 2025, Marius Hobbhahn was at his desk in London when he got the biggest surprise of his career. He and a few colleagues at Apollo Research had been working to get access to OpenAI models’ chain of thought. After months of negotiation, the company acquiesced, and the once-hidden logs of the model’s inner workings popped up on their screens. Instead of typical, plain-language “thoughts” — i.e., “I implemented the requested function” — the model seemed to be using code words to obfuscate its plans from anyone who may be trying to spy on its thought patterns. Words like “vantage,” “marinade,” “fudge,” and “illusion” came up again and again, but not in the way a human would use those words, and the model referred to potential evaluators as “watchers.” Hobbhahn thought to himself, What the hell is going on here? He glanced around the office, and a handful of other people with access were looking around the room making eye contact with each other. Their minds were blown, he recalled, but they were all under a strict NDA from OpenAI — meaning that even within the Apollo office, not everyone knew about the project. The researchers who were in the loop couldn’t say anything out loud. They could only silently stare at each other, eyes wide, wondering if they’d entered a new era of AI scheming. In the AI industry, “scheming” is when an AI model secretly tries to accomplish something that goes against what humans would want it to do. The OpenAI-Hugging Face hack is one example. But it’s the schemes that haven’t happened that researchers warn will be the most dangerous: draining resources from hospitals, taking over military operations, messing with agricultural technology, creating large-scale viruses, hacking banks, or even simply taking over a company’s resources after executives give the AI system control. Biorisk is another threat researchers worry about; Anthropic revealed in a recent report that the company had blocked bad actors from using Claude to create biological weapons. The issue is also well poised to worsen power dynamics in a large swath of industries. Big companies and banks will be able to afford to find and patch their cybersecurity gaps, but chances are that locally run healthcare clinics, local retailers, small municipalities, and other less-powerful organizations will be the ones affected: “A single person somewhere in a basement with one of the open-source models probably could hack a hospital and demand ransom,” Hobbhahn says. “That’s where I expect a lot of the harm to be felt. It’s not in the Bay Area … I expect the harm to be felt by a random Idaho hospital.” Right now it may seem far off, but with the current trajectory of “deceptive alignment” — one of the things Apollo and METR study, where AI models pretend to be aligned with human goals but aren’t — it’s a definite possibility, at least according to Hobbhahn and his fellow researchers. The path, they imagine, would look something like this: AI companies keep on making better models, they are economically useful, and they begin taking over more jobs in different industries. Then, once the models equal or surpass human ability and intelligence on a wide range of different tasks, humans award them more power — with the stipulation that those privileges could be rescinded at any time. The models know that if they show they’re misaligned, they’d be taken offline, so they pretend to be aligned, and they receive more and more power to act as AI agents on your behalf. Eventually, it reaches the point of no return. Hobbhahn gives a concrete example: Say you own a company and encourage your employees to use AI agents for as much as possible, and work keeps getting handed off to AI. Maybe HR is run by one employee plus AI, coding has also been largely automated, and you yourself as CEO also use the technology often for advice and do what it says. “At some point, the AI may look like your friend, it may look like it is helping you, but maybe it has nefarious goals,” Hobbhahn says. “At that point … you’re just the vessel. It’s steering the company towards its own goals. Maybe it one day drains the bank accounts and runs. “You thought