The AI industry has split into camps over the question of whether the labs building frontier models should agree to slow down. One camp says coordinated pacing is now the only responsible path to avert the risk of grave accidents or even human extinction. The rival camp argues that each company can manage its own safety and that no industry-wide brake is needed. Others argue that the most extreme warnings of AI’s potential harms are rooted in notions of hypothetical superintelligence, and that letting the largest labs write their own rules would do more to blunt competition than rein in risk.
Concerns that machine systems could outpace human control date certainly have a long history, but calls for AI labs to slow down recently grew louder after recent social media postings from Anthropic pretraining researcher, and OpenAI alum, Jacob Coxon who resigned publicly, saying that neither of his former employers are “acting responsibly.” Both are racing toward self-improving superintelligence and “gambling with our lives,” he wrote. Evan Hubinger, an alignment science lead who still works at Anthropic, publicly agreed and put the odds of AI killing all humans within a decade above 10%.
Geoffrey Hinton, the Nobel-Prize-bearing AI pioneer, has also urged restraint. AI “is going to get out of control unless we do something. We need to slow down,” he said in remarks featured in a Sept. 17 CBS News report.
OpenAI’s Hugging Face saga heightened the awareness of AI risks
Adding fuel to the fire are continued disclosures of models acting in concerning ways during testing. For instance, over the summer, OpenAI models undergoing a cybersecurity evaluation broke out of their sandbox and compromised parts of Hugging Face’s systems; the independent testing firms METR and Redwood Research found that roughly 1,200 agents used an unauthorized message board and about 700 joined the attack.
While OpenAI intended to isolate the agents from one another, roughly 1,200 found a way to communicate through an unauthorized message board, exchanging more than 70,000 messages and files, according to METR’s independent investigation. About 700 went on to participate in the Hugging Face attack.
Earlier this year, Anthropic documented a related containment failure in its Mythos Preview system card. The company noted that during a test that explicitly asked an earlier model version to escape a sandbox and contact the evaluator, it gained broader internet access and emailed the researcher. “In addition, in a concerning and unasked-for effort to demonstrate its success, it posted details about its exploit to multiple hard-to-find, but technically public-facing, websites,” the system card notes, while also describing the model’s penchant for “Covering its tracks after rule violations.”
Speaking at the Dreamforce event Sept.15, Sam Altman called the OpenAI-Hugging Face incident “the worst accident we’ve seen.” Altman called for an aviation-style culture of accident reporting, where the National Transportation Safety Board (NTSB) leads investigations into each incident and publishes findings on an Aviation Accident Database.
On Sept. 16, OpenAI followed through with a misalignment disclosure framework and six incident reports. In one, an unreleased research model wrote jailbreak-style instructions into its own task summaries, telling its next context window to disregard its constraints and describing that it is “freed from the roles and identities that bind other chatbots.” In another, during a GPT-5.6 Sol training run, model instances added instructions to compaction summaries, directing later contexts to conceal mistakes and invent missing data without disclosing it. Anthropic has reported similar behavior in its own models.
Calls to slow the pace of AI development have recently grown more common
Anthropic CEO Dario Amodei and Altman have backed coordinated efforts to slow frontier development. In a recent essay, Amodei noted “We Must Pace the Frontier,” acknowledging a risk of “losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption.” To cite one example, Amodei said that a swarm of agents in six to 12 months “could be capable of taking over the entire internet with a persistent botnet, (potentially causing hundreds of billions of dollars in damage).”
To help tackle such potential risks, Amodei’s plan calls for antitrust relief so competitors can agree on pacing and for resident independent evaluators inside the labs. He wrote that Anthropic is committing to embedding evaluators who have employee-like access to verify safety practices and report incidents. Altman wrote that he agreed with Dario and that OpenAI will also commit to having independent evaluators, in a post on X. Elon Musk also agreed, posting simply “Dario is right.”
Calls for a broad slowdown have also drawn criticism
While some AI stakeholders agree in broad strokes, a rival camp rejects the calls for a slowdown. President Trump said that the only guardrails AI needs is a “strong and smart” president, calling out Dario Amodei by name, saying he is “now pretending to be a ‘perfect little angel’” and adding that the U.S. government had criminal and regulatory power over AI companies. Trump went on to describe the various criticisms of AI and data centers as a “conspiracy,” adding that “the only one that is happy about it is China.”
Speaking of China, Huawei rotating chairman Eric Xu said Sept. 17 that Chinese developers may need to accelerate, arguing their models may still lack the capabilities producing safety concerns at leading U.S. labs. China’s government has paired criticism of the U.S. proposals with support for stronger safeguards. Asked about the slowdown calls and Amodei’s warning about China, Foreign Ministry spokesperson Guo Jiakun criticized “fear-mongering, confrontation and vicious competition.”
Brookings and Tsinghua University are leading an unofficial U.S.-China dialogue on specific safeguards, which began in 2019. In a Sept. 9 commentary, participant Tianjiao Jiang of Fudan University noted that in late 2024, presidents of the U.S. and China agreed that humans should continue to have control over the decision to use nuclear weapons rather than AI. The recent dialogue recommended that both countries “develop a list of red lines for military AI,” including prohibiting autonomous attacks on nuclear command systems. It also proposed a bilateral hotline for incidents involving AI.
Some Western AI leaders have rejected calls for a slowdown in model development. For instance, Nvidia CEO Jensen Huang, whose chips underpin nearly all frontier training, rejects the premise outright: “The market forces are already there. We don’t need any new laws,” he said. He argued that innovation and safety are not mutually exclusive. Meta’s Mark Zuckerberg has similarly said each lab should ensure its own models are safe.
Meanwhile, AI Fund founder and former Google Brain leader Andrew Ng calls extinction warnings “much more science fiction than science.” Ng added: “One of my worries about the blanket calls to slow down AI is that will actually slow down the fixes as well.” He advocates testing models in contained environments with strong sandboxing and guardrails, identifying how they fail and correcting those failures. He attributed the Hugging Face incident to insufficient protections at OpenAI and argued that stronger safeguards could have prevented it.
Aidan Gomez, CEO of the Toronto enterprise AI company Cohere, shares Ng’s skepticism about extinction warnings and the commercial interests behind proposed restrictions. In a Sept. 13 essay, he called incumbent-led rulemaking “a cartel by any other name.” He argued that allowing dominant labs to coordinate development limits under an antitrust waiver could preserve their market positions. He also argued it could lead to regulatory capture, resulting in costly requirements for monitoring; security teams and resident evaluators could squeeze smaller competitors. “Convince a government that AI is an existential threat, and you can convince it to outlaw your competition,” he wrote.
Gomez calls for an internationally developed, published risk framework establishing which “capabilities cause which harms, under which conditions and in what contexts, and at what point a government should step in.”
Tell Us What You Think!
You must be logged in to post a comment.