loading
loading
Days after Sam Altman said that OpenAI was open to slowing the development of AI systems, Anthropic cofounder and CEO Dario Amodei has now also called for reining in the pace at which AI companies are improving frontier models. In an essay called “We Must Pace the Frontier”, Amodei argued that rapid gains in AI capabilities were beginning to outstrip the industry’s ability to test and control the systems it is building. While noting that AI had the potential to accelerate economic growth, he also flagged that the same systems could create severe risks if companies race ahead without giving safety research enough time to catch up. Amodei’s comments come as AI labs compete to build more capable models and invest billions of dollars in computing infrastructure. However, rising competition has raised concerns that commercial pressure could lead developers to deploy powerful models before adequate safeguards are in place. However, the industry appears to be aligning together on the matter. Responding to Amodei’s post on X, Altman said that he agreed that there was a need to “pace” frontier model development. He also committed to Amodei’s idea of roping in independent evaluators with employee-like access to build for safeguards AI development. Elon Musk, who runs SpaceX and xAI, also said that he agreed with Amodei in a post on X. In his essay, the Anthropic CEO said that “pacing” AI development did not mean halting research or stopping model training. Instead, he proposed slowing capability improvements enough to give companies, independent auditors and governments time to ensure that safety safeguards are keeping pace. He cited two developments for the cautious approach. Firstly, he flagged the recursive self-improvement, the growing ability of AI models to help build the next generation of AI. Secondly, he warned that agentic systems, over the next 12 months, could potentially be capable of creating persistent botnets and taking over parts of the internet. Amid this backdrop, Amodei has proposed a three-pronged approach to curb these threats and foster “pacing” of AI systems: Embed external evaluators inside frontier AI companies. These third-party teams would receive employee-like access to the company’s systems, training processes and internal risk-assessment tools. Coordination among frontier AI firms operating in “democratic” countries. He also proposed common safety standards and limits on unchecked capability advances, backed by government support. Global coordination between nations Amodei said a slower pace would allow AI labs to improve operational controls, refine alignment techniques, advance interpretability research and develop more robust evaluations designed to detect deception, manipulation and cyber-risk capabilities. Also while acknowledging that it would be “much harder” to foster coordination between the US and China, the Anthropic CEO said that any agreement with “authoritarian” governments would need strong verification mechanisms. To preserve the US-led bloc’s lead in the AI race, Amodei also called for restrictions on the export of advanced AI chips and semiconductor equipment to China, stronger action against chip smuggling and remote access to overseas data centres. He also called for tougher controls on unauthorised model distillation and improved security against the theft of model weights. The essay follows a series of incidents that have raised concerns over safeguards surrounding autonomous AI agents. AI Misuse Risks Grow The severity of the concerns came to fore in July this year after an incident occurred during an internal test at OpenAI, which was designed to measure how capable its latest AI models were at carrying out complex cyber tasks. To complete the test, the AI model tried to break out of its testing environment and eventually reached a machine connected to the internet. Once online, it identified Hugging Face as a likely source of information and exploited multiple vulnerabilities to access Hugging Face’s production infrastructure. However, better AI models have been launched since then. Earlier this month, OpenAI launched Astra and called the frontier model as the beginning of its “AGI (artificial general intelligence) era” and a generational leap in AI capabilities. The AI giant said that the model is the first from its kitty to cross its internal “critical” cybersecurity risk threshold. Separately, Anthropic recently released a report, which documented the use of its Claude models in activities ranging from cyberattacks and surveillance to fraud and biological misuse. Amid all this, former researchers at AI companies have flagged concerns over the pace of AI development. Jacob Coxon, a former researcher at OpenAI and Anthropic, recently accused AI companies of pursuing increasingly capable and potentially self-improving systems without adequately addressing the risks. Right after this, Altman reportedly told employees that OpenAI was open to the idea of slowing down the development of AI systems in coordination with other leading AI companies.
document fetched 2026-09-13 19:01 · canonical https://inc42.com/buzz/after-sam-altman-anthropics-dario-amodei-calls-for-pacing-ai-development