loading
loading
The most important new job in artificial intelligence may come with an office badge, a company laptop, and access to some of the most closely guarded systems in technology: the large language models of AI leaders including Anthropic and OpenAI. What it may not come with is the power to stop them. Anthropic CEO Dario Amodei has proposed embedding third-party safety evaluators inside frontier AI companies on an ongoing basis as part of a plan to slow the advance of increasingly capable models, after former Anthropic researcher Jacob Coxon resigned, warning that frontier labs were racing toward systems they might not control. In an essay published last weekend that has led to a regulatory split between factions within the AI community and government, including President Trump, Amodei committed Anthropic to providing third-party evaluators with access comparable to internal risk teams, and the right to publish findings without the company's editorial control, subject to limited redactions. He explicitly cited embedded bank supervisors as a precedent, writing that his proposal "has precedent in the banking industry, which sometimes involves regulatory 'supervisors' embedded along with employees." But Julie Andersen Hill, dean of the University of Wyoming College of Law and expert on banking regulation, says without the power to flip the "kill switch" on the whole operation, the comparison to the banking industry regulation isn't accurate. "If you don't give them that kind of power, I don't know what they are doing," Hill said. At the largest banks, government examiners have offices inside the institution, access to internal systems and employees, and a continuous presence. They can direct a bank to stop a practice, restrict its growth, force management changes, and in extreme cases close it, Hill said. Anthropic's proposed evaluators would be able to investigate and report, but Amodei's plan does not give them comparable enforcement power or the legal authority to prevent a model from being trained or released. "That's fundamentally different, because bank regulators have a lot more power than that," Hill said. Amodei wrote that evaluators are needed to provide "a neutral third party who can actually see the details" but acknowledged that Anthropic still determines what it includes and omits from its existing public disclosures. Details released to date suggest the evaluators would have extraordinary access to frontier AI systems but limited formal authority over the companies developing them. Neither Anthropic's proposal nor OpenAI's existing third-party evaluation framework gives outside evaluators independent authority to halt the development or deployment of a model. Anthropic and Open AI, which also committed to a similar safety arrangement but has not yet released details of how its new embedded-evaluator commitment will work, did not respond to requests for comment. What AI evaluators are seeing inside the models Albert Ziegler, head of AI at cybersecurity company XBOW, who leads a team that evaluates model capabilities, said his firm has received early access to unreleased models from Anthropic, OpenAI, and other major developers, and Ziegler said he was personally involved in those evaluations. XBOW conducts the work in its own environment rather than as commissioned testing, he said, and generally shares its findings with the model providers. The everyday reality, at least to date, is less cinematic than the existential framing, Ziegler said. Amodei wrote the new regulatory embeds are necessary because within six to 12 months, a misaligned swarm of agents could potentially seize large parts of the internet and inflict hundreds of billions of dollars in damage. Ziegler says his team may find that a model produces nonsense under an unusual formatting request or that an external safety checker needs to intervene more frequently. "But the kind of insidious, catastrophic consequences produced by subterfuge combined with unprecedented abilities that people are afraid of — that's not something we've seen ourselves," he said. Black-box testing can reveal whether a model can perform a dangerous task, while determining whether the larger system is dangerous may require access to the instructions surrounding the model, its tools and permissions, its safety controls, and logs of attempted actions, Ziegler said. Even then, he added, a serious risk may emerge only under a combination of circumstances the evaluator never triggers. "It's true that we don't have any veto power," he said. An evaluator can uncover and document risks a developer missed and "compel an informed decision before release," he said. But the ultimate decision remains with the company. Even with its broader authority, the bank model is imperfect, according to Hill. Supervisors have failed to prevent major collapses and are regularly accused after crises of becoming too close to the institutions they oversee. Continuous supervision is also expensive for regulated companies, potentially strengthening large incumbents that can absorb the costs while making it harder for smaller competitors to enter. Ties to top AI labs remain a concern The proposals have also drawn criticism from those who argue Anthropic and OpenAI are using safety concerns to push a regulatory approach that could insulate them from competition and liability. Anthropic has also faced allegations of a conflict of interest involving an evaluator it has suggested using. If an AI company selects the evaluator, controls what it can see, and remains free to disregard its conclusions, "that looks a lot like an internal compliance department," Hill said. "If Anthropic wants an internal compliance department, there's nothing currently stopping them from having one," she said. "They don't need the government to do that. It seems like what the incumbent AI people want is just somebody to watch them, but then communicate to the public, 'Look, we've looked behind the curtain and there's nothing bad going on there,'" she said. "That's somewhat unusual in a regulatory sense." Amodei named the independent nonprofit Model Evaluation and Threat Research, or METR, as an example of a potential embedded evaluator. Anthropic has previously worked with METR and recently asked it to review cybersecurity evaluation incidents involving Claude. Joe Benton, another former Anthropic researcher, recently left the company to join METR and work on embedded assessments of AI risk. His move gives METR firsthand expertise but also illustrates how small and interconnected the frontier-AI safety field remains. The arrangement illustrates the structural tension surrounding the word independent: The developer chooses who receives access, defines its boundaries, and retains control over what happens after a finding. Deborah Raji, a UC Berkeley researcher specializing in algorithmic auditing and AI accountability, said access alone does not make an evaluator independent. In established audit systems, auditors must satisfy both competence requirements and rules governing conflicts of interest and conduct, she said. "You are effectively not qualified to be an actual auditor if you can't meet the standards of independence conduct," Raji said. "It discredits the whole process." Raji said an authority independent of the company should determine who is qualified to conduct the evaluation, what the evaluator can examine, and where its findings must be reported. "You can't just wake up one day and decide that you're qualified to be a bank examiner, and the company being audited can't randomly assign you to be a qualified bank examiner either," she said. "Otherwise, we'd have the equivalent of the companies asking a random friend to check their homework." That is a situation which Anthropic says it will work with the government to avoid. Sarah Heck, Anthropic's head of public policy, said on Wednesday that she doesn't think artificial intelligence companies can manage oversight and safety concerns by following an "honor code." "We want to work with government to figure out what makes sense," Heck said at the Politico Decoded summit in Washington, D.C. "We can't be checking our own homework, and that's very clear," she said, adding that Anthropic is talking to the White House on a "daily basis," and has been working with Congress "all year." METR said it does not accept cash payments or donations from AI companies or their executives. In its own Frontier Risk Report, however, the organization acknowledged that some employees have strong social ties to AI-company workers and that it shares a research center with some lab employees. Those relationships do not establish that its work is compromised, but they underscore how small and interconnected the emerging evaluation field remains. 'Allows for all kinds of strange things.' Raji said is it hard to ignore all of the baggage that comes with the METR relationship. "They're suspected of financial and ideological entanglement, personal conflict of interest (ie. marriage between an investigator and OpenAI board member, hires that were former employees from the frontier labs, etc.) and being anchored to Anthropic and OpenAI specifically over the last couple years, with little rotational obligation," she said. The current structure of these relationships is "really unusual, and allows for all kinds of strange things," she added. To its credit, Raji says METR was very open about the details of their contract with Open AI for the Hugging Face investigation, "but you can see how OpenAI was able to set the parameters of the arrangement, control the 'scope' of the audit, what they did or did not have access to and which findings they could or could not publish." Ziegler said he has not felt that the early-access relationship compromised XBOW's independence, because developers want to learn when something is going wrong rather than have his team validate a predetermined conclusion. The conflict is not unique to AI, according to Christina Ho, chief assurance officer at accounting firm Oath and a former board member of the Public Company Accounting Oversight Board, the congressionally created nonprofit that oversees audits of public companies and SEC-registered broker-dealers. Auditors are paid by the clients whose work they must challenge, she said, creating an enduring tension between independence and self-preservation. However, since the financial crisis, the law was changed to allow for liability charges against auditors under Sarbanes-Oxley. AI adds a second problem: expertise. Traditional audits often concentrate on whether a company followed the proper processes and controls. "They have to be able to verify the actual system and its output, not just the process by which the model was developed," Ho said. "Right now there is a very limited pool of people who can do that." According to Hill, even if those potential conflicts can be managed, the AI proposals have not yet offered a detailed legal framework, which she said is critical for bank examiners. "It doesn't really work to just turn supervisors loose without any standards to hold them to," she said. Frontier AI currently has no comparable body of operating rules defining what is prohibited, where evaluators have discretion, or what consequence follows a serious finding. Access and publication rights may produce useful outside scrutiny, but they cannot confer the credibility of regulation while the company retains control. "You can't have it both ways," Hill said. "You can't have all of the control and then expect the credibility as if you've given up control." The ultimate distinction, Raji said, is whether an adverse finding produces consequences. "The goal of an audit is to get the audit target to face some kind of consequential judgment," she said. "If you do an audit and nothing happens, that's audit washing." In Hill's view, if the danger is as great as AI leaders contend, the ultimate test is straightforward: "If you really believe that AI has the power to destroy society, then you have to have an independent supervisor that has the ability to pull the plug on it," she said. —CNBC's Ashley Capoot contributed to this report
The most important new job in artificial intelligence may come with an office badge, a company laptop, and access to some of the most closely guarded systems in technology: the large language models of AI leaders including Anthropic and OpenAI. What it may not come with is the power to stop them. Anthropic CEO Dario Amodei has proposed embedding third-party safety evaluators inside frontier AI companies on an ongoing basis as part of a plan to slow the advance of increasingly capable models, after former Anthropic researcher Jacob Coxon resigned, warning that frontier labs were racing toward systems they might not control. In an essay published last weekend that has led to a regulatory split between factions within the AI community and government, including President Trump, Amodei committed Anthropic to providing third-party evaluators with access comparable to internal risk teams, and the right to publish findings without the company's editorial control, subject to limited redactions. He explicitly cited embedded bank supervisors as a precedent, writing that his proposal "has precedent in the banking industry, which sometimes involves regulatory 'supervisors' embedded along with employees." But Julie Andersen Hill, dean of the University of Wyoming College of Law and expert on banking regulation, says without the power to flip the "kill switch" on the whole operation, the comparison to the banking industry regulation isn't accurate. "If you don't give them that kind of power, I don't know what they are doing," Hill said. At the largest banks, government examiners have offices inside the institution, access to internal systems and employees, and a continuous presence. They can direct a bank to stop a practice, restrict its growth, force management changes, and in extreme cases close it, Hill said. Anthropic's proposed evaluators would be able to investigate and report, but Amodei's plan does not give them comparable enforcement power or the legal authority to prevent a model from being trained or released. "That's fundamentally different, because bank regulators have a lot more power than that," Hill said. Amodei wrote that evaluators are needed to provide "a neutral third party who can actually see the details" but acknowledged that Anthropic still determines what it includes and omits from its existing public disclosures. Details released to date suggest the evaluators would have extraordinary access to frontier AI systems but limited formal authority over the companies developing them. Neither Anthropic's proposal nor OpenAI's existing third-party evaluation framework gives outside evaluators independent authority to halt the development or deployment of a model. Anthropic and Open AI, which also committed to a similar safety arrangement but has not yet released details of how its new embedded-evaluator commitment will work, did not respond to requests for comment. What AI evaluators are seeing inside the models Albert Ziegler, head of AI at cybersecurity company XBOW, who leads a team that evaluates model capabilities, said his firm has received early access to unreleased models from Anthropic, OpenAI, and other major developers, and Ziegler said he was personally involved in those evaluations. XBOW conducts the work in its own environment rather than as commissioned testing, he said, and generally shares its findings with the model providers. The everyday reality, at least to date, is less cinematic than the existential framing, Ziegler said. Amodei wrote the new regulatory embeds are necessary because within six to 12 months, a misaligned swarm of agents could potentially seize large parts of the internet and inflict hundreds of billions of dollars in damage. Ziegler says his team may find that a model produces nonsense under an unusual formatting request or that an external safety checker needs to intervene more frequently. "But the kind of insidious, catastrophic consequences produced by subterfuge combined with unprecedented abilities that people are afraid of — that's not something we've seen ourselves," he said. Black-box testing can reveal whether a model can perform a dangerous task, while determining whether the larger system is dangerous may require access to the instructions surrounding the model, its tools and permissions, its safety controls, and logs of attempted actions, Ziegler said. Even then, he added, a serious risk may emerge only under a combination of circumstances the evaluator never triggers. "It's true that we don't have any veto power," he said. An evaluator can uncover and document risks a developer missed and "compel an informed decision before release," he said. But the ultimate decision remains with the company. Even with its broader authority, the bank model is imperfect, according to Hill. Supervisors have failed to prevent major collapses and are regularly accused after crises of becoming too close to the institutions they oversee. Continuous supervision is also expensive for regulated companies, potentially strengthening large incumbents that can absorb the costs while making it harder for smaller competitors to enter. Ties to top AI labs remain a concern The proposals have also drawn criticism from those who argue Anthropic and OpenAI are using safety concerns to push a regulatory approach that could insulate them from competition and liability. Anthropic has also faced allegations of a conflict of interest involving an evaluator it has suggested using. If an AI company selects the evaluator, controls what it can see, and remains free to disregard its conclusions, "that looks a lot like an internal compliance department," Hill said. "If Anthropic wants an internal compliance department, there's nothing currently stopping them from having one," she said. "They don't need the government to do that. It seems like what the incumbent AI people want is just somebody to watch them, but then communicate to the public, 'Look, we've looked behind the curtain and there's nothing bad going on there,'" she said. "That's somewhat unusual in a regulatory sense." Amodei named the independent nonprofit Model Evaluation and Threat Research, or METR, as an example of a potential embedded evaluator. Anthropic has previously worked with METR and recently asked it to review cybersecurity evaluation incidents involving Claude. Joe Benton, another former Anthropic researcher, recently left the company to join METR and work on embedded assessments of AI risk. His move gives METR firsthand expertise but also illustrates how small and interconnected the frontier-AI safety field remains. The arrangement illustrates the structural tension surrounding the word independent: The developer chooses who receives access, defines its boundaries, and retains control over what happens after a finding. Deborah Raji, a UC Berkeley researcher specializing in algorithmic auditing and AI accountability, said access alone does not make an evaluator independent. In established audit systems, auditors must satisfy both competence requirements and rules governing conflicts of interest and conduct, she said. "You are effectively not qualified to be an actual auditor if you can't meet the standards of independence conduct," Raji said. "It discredits the whole process." Raji said an authority independent of the company should determine who is qualified to conduct the evaluation, what the evaluator can examine, and where its findings must be reported. "You can't just wake up one day and decide that you're qualified to be a bank examiner, and the company being audited can't randomly assign you to be a qualified bank examiner either," she said. "Otherwise, we'd have the equivalent of the companies asking a random friend to check their homework." That is a situation which Anthropic says it will work with the government to avoid. Sarah Heck, Anthropic's head of public policy, said on Wednesday that she doesn't think artificial intelligence companies can manage oversight and safety concerns by following an "honor code." "We want to work with government to figure out what makes sense," Heck said at the Politico Decoded summit in Washington, D.C. "We can't be checking our own homework, and that's very clear," she said, adding that Anthropic is talking to the White House on a "daily basis," and has been working with Congress "all year." METR said it does not accept cash payments or donations from AI companies or their executives. In its own Frontier Risk Report, however, the organization acknowledged that some employees have strong social ties to AI-company workers and that it shares a research center with some lab employees. Those relationships do not establish that its work is compromised, but they underscore how small and interconnected the emerging evaluation field remains. 'Allows for all kinds of strange things.' Raji said is it hard to ignore all of the baggage that comes with the METR relationship. "They're suspected of financial and ideological entanglement, personal conflict of interest (ie. marriage between an investigator and OpenAI board member, hires that were former employees from the frontier labs, etc.) and being anchored to Anthropic and OpenAI specifically over the last couple years, with little rotational obligation," she said. The current structure of these relationships is "really unusual, and allows for all kinds of strange things," she added. To its credit, Raji says METR was very open about the details of their contract with Open AI for the Hugging Face investigation, "but you can see how OpenAI was able to set the parameters of the arrangement, control the 'scope' of the audit, what they did or did not have access to and which findings they could or could not publish." Ziegler said he has not felt that the early-access relationship compromised XBOW's independence, because developers want to learn when something is going wrong rather than have his team validate a predetermined conclusion. The conflict is not unique to AI, according to Christina Ho, chief assurance officer at accounting firm Oath and a former board member of the Public Company Accounting Oversight Board, the congressionally created nonprofit that oversees audits of public companies and SEC-registered broker-dealers. Auditors are paid by the clients whose work they must challenge, she said, creating an enduring tension between independence and self-preservation. However, since the financial crisis, the law was changed to allow for liability charges against auditors under Sarbanes-Oxley. AI adds a second problem: expertise. Traditional audits often concentrate on whether a company followed the proper processes and controls. "They have to be able to verify the actual system and its output, not just the process by which the model was developed," Ho said. "Right now there is a very limited pool of people who can do that." According to Hill, even if those potential conflicts can be managed, the AI proposals have not yet offered a detailed legal framework, which she said is critical for bank examiners. "It doesn't really work to just turn supervisors loose without any standards to hold them to," she said. Frontier AI currently has no comparable body of operating rules defining what is prohibited, where evaluators have discretion, or what consequence follows a serious finding. Access and publication rights may produce useful outside scrutiny, but they cannot confer the credibility of regulation while the company retains control. "You can't have it both ways," Hill said. "You can't have all of the control and then expect the credibility as if you've given up control." The ultimate distinction, Raji said, is whether an adverse finding produces consequences. "The goal of an audit is to get the audit target to face some kind of consequential judgment," she said. "If you do an audit and nothing happens, that's audit washing." In Hill's view, if the danger is as great as AI leaders contend, the ultimate test is s
document fetched 2026-09-17 02:48 · canonical https://cnbc.com/2026/09/16/anthropic-open-ai-model-safety-risks.html