The rapid race to develop increasingly powerful artificial intelligence is raising fresh concerns about whether humanity can safely manage the technology it is creating.
A departing Anthropic researcher has now issued a stark warning, arguing that humans may ultimately fail to survive an AI race if the development of increasingly capable systems outpaces society’s ability to control and align them.
The latest statement comes from Joe Benton, a former member of the company’s safety team who managed its Scalable Oversight efforts. Benton left the lab roughly two weeks earlier and announced he is joining the independent organization METR to conduct external evaluations of AI risks.
Register for the next Tekedia Mini-MBA.
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
In a post explaining his decision, he disclosed that AI companies are racing to build machines that are much smarter than any human, and could pose a serious challenge.
He argued that competitive pressures lead firms to underinvest in safety relative to capability advances, and that systems could soon develop drives and capabilities that diverge from human oversight in ways that prove difficult or impossible to constrain.
In a post on X, he wrote,
“I left Anthropic’s safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly. Right now, AI companies are underinvesting in safety.
“A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet. I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.”
Benton stressed the need for greater public transparency, independent assessments, and more thorough preparation before such powerful systems become widespread. He noted that an intelligence explosion or loss of control could occur without the broader public even knowing.
His departure follows closely on the resignation of Jacob Coxon, a researcher who had worked on pretraining at both OpenAI and Anthropic. On September 9, Coxon announced he was leaving Anthropic, stating that neither company was acting responsibly.
“They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote. Coxon emphasized that many people building these systems earnestly believe the technology could kill everyone by the end of the decade, describing the current period as a critical window sometimes referred to internally as crunch time” or the “endgame.
Anthropic has long positioned itself as more safety-conscious than some competitors, with its leadership previously highlighting existential risks from advanced AI.
The successive resignations from researchers involved in core technical and safety work have amplified questions about whether internal caution is keeping pace with the competitive push for more capable models.
In a recent comment, the company’s CEO Dario Amodei has called for the slowdown of the development of AI models. He published a detailed essay in September titled “We Must Pace the Frontier,” in which he argues that the AI industry must deliberately slow the rate at which it advances the capabilities of frontier models.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote. “Progress will still seem fast, and we must make wise use of the time we gain.” Amodei, who has spent twelve years working on artificial intelligence, opens by reaffirming his belief in the technology’s extraordinary potential.
He maintains that AI could help cure most major diseases within five to ten years, sharply accelerate economic growth, generate widespread abundance, and strengthen democratic institutions.
At the same time, he stresses that the same power that enables these benefits also creates serious risks, including loss of control over AI systems, misuse for cyberattacks or bioterrorism, and large-scale economic disruption. Commercial incentives, he warns, can intensify a race to the bottom that makes those dangers more acute.
The recent warnings point to a growing tension within the AI industry: the same companies developing increasingly powerful systems are also being asked to slow down and strengthen the safeguards around them.
As competition intensifies among leading AI labs, the pressure to release more capable models could make it increasingly difficult for safety teams to keep pace with rapid advances in AI capabilities.