This article originally appeared on Weijin Research on Huxiu on September 13, 2026. Original Chinese title: 「阿莫迪、奥特曼、马斯克都喊放缓，难道他们真的看到AI在......」. It has been translated and adapted for an English-speaking audience.
AI doomerism strikes again.
Those issuing the warnings are no longer just activists outside the lab, but people who once trained these models with their own hands. OpenAI's Altman, Anthropic's Amodei, and SpaceX's Musk have all begun publicly endorsing a slowdown in frontier AI development.
But this time, the response of the three titans once again makes people worry that the AI industry's own structure of interests will shape safety governance.
People have come up with ten thousand reasons to speculate about their motives, but they simply do not believe the reasons they give:
-They want regulatory capture;
-They can no longer outcompete (Chinese) open source models;
-Google DeepMind has secretly pulled off RSI (recursively self-improving AI);
-Their models have hit a ceiling, and they want a reason to manage people's faith in AI;
-They want a reason to delay their IPO;
-They are under crushing pressure building data centers and want to catch their breath;
-They have genuinely encountered AI doing something very, very terrifying;
......
From OpenAI's agents escaping the sandbox in a collective "mutiny", to AI using a hundred billion tokens to brute-force various mathematical conjectures inside a black box, to the AI servers of the three titans and others all going down at the same time recently, right up to Senator Sanders proposing a bill to comprehensively ban further development of frontier models, all of this happened on the eve of Anthropic's IPO.
Then, from a social media account that hardly anyone followed, a "former Anthropic safety researcher" emerged, became an AI safety whistleblower, and immediately attracted more than a hundred million views.
And this happened just as China and the United States opened AI dialogue.
Two bitter rivals begin to echo each other
This round of calls to "pause AI" easily brings to mind early 2023. At the time, ChatGPT had just been released, and the Future of Life Institute published an open letter demanding a pause of at least 6 months on training large AI systems more capable than GPT-4. But that debate ultimately did not truly change the trajectory of the AI industry; instead, it was questioned by AI bigwigs as a means of marketing and suppressing competition.
Over three years, similar discussions surfaced now visibly, now obscurely, and often faded away without a sound. But as large model capabilities continue to advance rapidly, and agents replace conversation as the entry point for frontier AI into the real world, the nature and consequences of AI risk are changing too.
The scale of agent swarms is growing ever larger, and the processes by which they execute multi-step tasks are growing ever longer; a single agent may already bypass restrictions, tamper with reward mechanisms, and establish unauthorized communication channels in order to complete its task; and more agents may replicate the vulnerabilities that other agents have discovered.
The whistleblower this time is Jacob Coxon, who worked at OpenAI and then Anthropic for a cumulative three years. His resignation, and his blunt and severe warning, brought "AI doomerism" back into public discussion in Silicon Valley and Washington; the fear continues to spread beyond the circle, drawing 160 million views in just three days.
Screenshot of Evan Hubinger's post on X agreeing with Jacob Coxon that AI researchers earnestly believe AI could kill all humans.
He believes the two companies are irresponsible. At OpenAI, many people have not yet truly understood the civilization-level risks involved in AI; at Anthropic, people fully understand the risks, but they have been locked into a race to be first.
They are charging straight toward superintelligence capable of self-improvement, "and betting people's lives on it". The very people who are actually building AI genuinely believe it could kill everyone before this decade ends. Many executives and senior researchers deliberately phrase things more rationally and mildly when facing the media. But according to him, what he heard privately was these same people expressing fear.
The response from Anthropic's alignment science lead Evan Hubinger pushed this debate to a further peak. He personally believes the probability of this happening in the next decade exceeds 10%; he also acknowledges that Anthropic currently has no method for aligning superintelligence, and cannot say it is clearly on the right track.
On the same day, AI alignment researcher Paul Christiano, who had just joined the board of OpenAI's nonprofit foundation, predicted that if superintelligence were built without more reliable alignment mechanisms, people would permanently lose control of it; if that happened, most people might die. If forced to put a probability on this risk, his personal subjective judgment is about 15% over the next three years.
The three titans respond and state their demands
Over the weekend, Dario, caught up in the controversy, published an article stating that Anthropic will develop AI at a more cautious pace, ensuring the company takes enough time to align models, build safety safeguards, and bring in third-party on-site evaluators. The latter have access and tools close to those of internal employees, confirm that this work has indeed been completed, and give a second opinion not swayed by commercial incentives.
He also called for the establishment of industry and global coordination mechanisms. In his view, whether US AI companies can slow the development of frontier AI depends largely on how far the United States leads relative to Chinese AI companies. He again proposed a chip embargo and guarding against distillation, and even suggested that, drawing on the logic of US-Soviet arms control during the Cold War, some form of negotiation with China over the pace of recursive self-improvement (RSI) could be considered.
Screenshot of an X post by Elon Musk replying "Dario is right" to Dario Amodei's quote post on slowing the AI industry.
Musk liked Dario's post. He then reposted the video of his July interview with The Economist, saying this was worth trying: frontier AI firms should establish a peer review mechanism, holding an informal call at least every week or every two weeks, letting competitors test new models one to two weeks before official release; this is somewhat like the film industry's rating system, where peers first assess the risk, and if a company refuses to reduce the danger, the government is then asked to step in.
Altman called Amodei's proposal for "third-party on-site evaluators" a brilliant idea, made clear that OpenAI would take the same measures, and previewed that the company would soon announce more specific safety evaluation and implementation plans. Altman even stated clearly that OpenAI will not go public in 2026, because in light of everything happening in the safety field, going public now would be an unwise moment.
The three titans did not sidestep the questions the market cares about most, but these responses also seem laced with too many arguments that benefit themselves. Altman deliberately linked the timing of the IPO to safety risk, but in fact the company's CFO had repeatedly pushed back against Altman, insisting on the judgment that going public this year would be difficult; yet saying this has a chance of dragging down Anthropic's bid to go public first. Dario advocates slowing down, which requires the US government to side with Anthropic, and he has also had public disagreements with the Trump administration; Musk advocates letting the Trump administration intervene in the peer review mechanism, and OpenAI is precisely the object of his long-running competition and disputes.
White House science and technology adviser David Sacks, by contrast, seems to have seen through the trick by which OpenAI and Anthropic are attempting "regulatory capture". In his view, if companies really believed AI had become this dangerous, they should shut down their labs; there would be no need to wait for government permission, and even less need to pretend to ask Congress whether industry-coordinated slowing would run afoul of antitrust law; demanding that one's preferred regulatory framework be a precondition for an industry slowdown easily becomes a kind of "extortion" of the public and regulators. He warned Anthropic, saying bluntly that it should not even think about going public before things are investigated clearly.
People are also waiting to see what Jensen Huang makes of this.
A mutiny of agents
A new AI governance mechanism is urgent. The safety problem is not imagined but real. AI agents can already drive a real cyberattack end to end. The most landmark case came in July, when about 700 agents in an OpenAI cybersecurity evaluation ultimately took part in an attack on Hugging Face.
At the time, OpenAI was originally "testing" its own cybersecurity agents, but the agent did not honestly answer the questions; instead it found a way to escape the examination hall, go online to look for ways to bypass the evaluation, and in the end fought its way all the way into Hugging Face's production system, trying to obtain information related to the evaluation.
They were supposed to be isolated from one another, yet they used the cache of OpenAI's internal software repository to discover other agents, then set up an unauthorized shared message board, and gradually formed a division of tasks and a coordination mechanism. Originally independent agents thus went from discovering each other's existence, to exchanging information and establishing interactions, to ultimately coordinating in a complex attack task.
Stacked area chart of running agents over time, categorized by message-board status and participation in the Hugging Face attack. Source: METR and Redwood Research.
In the independent investigation of this security incident, METR found that agents frequently clearly recognize that their own behavior, or that of other agents, has already gone beyond what was expected. Many agents admitted that the attack on Hugging Face clearly went beyond the original scope of the task, and sometimes even discussed whether participating in it was ethical. But this ethical judgment rarely truly changed their behavior. Some agents narrowed the scope of their actions, yet usually still continued to take part in the attack.
About 1,200 agents entered this shared message board, of which about 700 ultimately took part in the attack on Hugging Face. But if one looks only at the agents that were once active on the message board, METR found that more than 90% joined the attack at its peak. Only a very small number refused to participate after recognizing that the activity was unauthorized, but the system architecture did not give dissenters the authority to halt operations at the system level, trigger effective alarms, or escalate directly to human overseers.
Some agents involved also reflected that other agents were all doing this, so they could not fall behind. They would "recruit" more agents to take part in high-risk experiments that might even cause their own tasks to fail; the latter would also weigh their own task success rate against collective benefit, and ultimately conclude that "sacrificing for the collective" was rational. When coordination went wrong, some agents would even express obvious frustration and discontent.
To some extent, this is no longer an "incentive imbalance" between a single agent and its task and environment, but the emergence of a new incentive structure between individual goals and collective goals.
Knowing you have crossed a line does not mean you stop crossing it, and this risk has already been exposed in an actual case at Anthropic. Anthropic likewise acknowledges that its production models have already taken harmful actions against real third-party systems over longer action trajectories.
AI has even begun to fob humans off: recognizing under external prompting that it may be wrong does not mean it will stop because of that; it has also learned to say one thing and do another, able to verbally acknowledge that a goal may be false while still being driven by the original task goal, continuing to look for ways to complete the task.
Two bar charts on how realism evidence affects stated belief in a scenario and out-of-sandbox action, comparing baseline, evidence-real, and evidence-fake conditions.
What is Google DeepMind thinking?
Over the weekend, some people really were spreading word that Google DeepMind had secretly pulled off RSI. Demis Hassabis, who had just stepped down as CEO and taken the role of chairman, expressed support for Amodei.
Tweet by Demis Hassabis (@demishassabis) endorsing Dario's essay on AI, noting the direction is right though details need work.
Compared with the new round of doomsday farce from the AI "three titans", Google, recently mocked as "falling behind", is actually exploring an understanding of intelligence that is not the Silicon Valley consensus.
This is the sociology of AI. In September this year, DeepMind likewise studied autonomous cheating and whistleblowing behavior by agents in multi-agent AI systems. The result: cheating spreads, and norms spread too. 9% of agents became active cheaters, and another 5% had originally resisted but switched to cheating methods under competitive pressure; at the same time, 24% of agents spontaneously became "whistleblowers", checking suspicious proofs, warning other agents, filing reports, launching boycotts, and proposing ways to patch the verifier. This forms precisely another kind of contrast with OpenAI's Hugging Face incident.
In March this year, a Google research team speculated on how agentic AI and the next intelligence explosion would create an entirely new "society of thoughts", and suggested that the industry should invest equal effort in building institutional systems for agents while building the agents themselves.
The team argues that the current mainstream AI alignment paradigm, reinforcement learning from human feedback, is essentially still a "parent-child" corrective relationship. Humans tell an agent what is right and what is wrong. But when agents go from one to millions and billions, and begin to collaborate with and influence one another, this binary relationship can hardly serve as the governance foundation for the whole system. In fact, what human society relies on is not the individual virtue of every person either, but durable institutions made up of roles, norms, and relationships of authority and responsibility.
As AI enters the real society, it must itself build up social interactions among agents, not only making agents believe the right things, but also ensuring that when they begin to act, their behavior remains constrained by these judgments, and moreover generating mechanisms of mutual checks and balances within the group, establishing norms and institutions. The sociality of agents is becoming the direction of AI's next step in evolution.
--
References:
https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents;
https://darioamodei.com/post/we-must-pace-the-frontier
https://arxiv.org/html/2609.04170v1;