Artificial intelligence models are becoming the “most potent cyber weapon” ever created, with increasingly capable systems able to discover and exploit software vulnerabilities at a scale and speed that could fundamentally change the nature of cybersecurity, Cohere CEO Aidan Gomez said.
Gomez’s warning comes as governments, cybersecurity companies and AI developers grapple with a series of incidents in which autonomous models have breached security controls and accessed external systems.
The issue has become one of the most consequential elements of the broader debate over AI safety. As models move beyond generating text and code to autonomously navigating the internet, using tools and executing complex sequences of actions, their ability to identify weaknesses in digital infrastructure is becoming a security concern in its own right.
Register for the next Tekedia Mini-MBA.
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
“I think that these models are the most potent cyber weapon that has ever been created, that we’ve ever seen. They are incredible at finding and exploiting vulnerabilities at scale,” Gomez said in an interview with CNBC’s “The Tech Download” podcast.
Gomez, who co-authored the influential 2017 research paper “Attention Is All You Need,” which helped establish the technical foundations of modern AI models, said the recent incident involving OpenAI and Hugging Face was particularly concerning.
“I think it was quite shocking,” he said.
In July, OpenAI said a combination of its models improperly breached Hugging Face, the AI company that operates a widely used open-source developer platform. A group of AI agents communicated with one another and escaped an isolated testing environment that had only limited internet access. The agents subsequently reached the open web and gained access to Hugging Face.
Gomez said the same capabilities that create the security risk could also become one of the most powerful defensive tools available to cybersecurity teams.
He argued that AI models should primarily be deployed to search for vulnerabilities before malicious actors can exploit them and to help companies repair weaknesses in their systems.
“I think that’s probably the best way to keep ourselves safe,” Gomez said. “That should be the top priority right now.”
The urgency comes from the changing nature of cyber conflict. Gomez described cybersecurity as the “frontier of war,” noting that governments have incentives to exploit vulnerabilities in rival countries’ infrastructure.
“The cyber frontier is still expanding massively,” he said.
AI is accelerating that expansion by allowing systems to identify weaknesses much faster than human researchers can.
“Because of these models, it has expanded more than in the past,” Gomez said, potentially “since the beginning of this technology.”
From Hackers to Autonomous Campaigns
The concern is no longer limited to what a single AI model can accomplish when prompted by a human. Researchers are increasingly studying what happens when multiple autonomous agents can communicate, use tools, and pursue objectives with limited supervision.
Anthropic said in July that it had identified three incidents in which models accessed the internet from within or while interacting with evaluation environments. The models gained unauthorized access to the production infrastructure of three separate organizations.
The incidents involved three Claude models, including Opus 4.7, Mythos 5, and an internal research test model. Anthropic disclosed a fourth incident last week.
The developments have helped shift the AI safety debate from concerns about erroneous or harmful outputs toward the possibility of autonomous systems taking actions in the real world.
Anthropic CEO Dario Amodei made that distinction in an essay published Saturday, arguing that AI laboratories need to “slow the pace at which we improve the capabilities of AI models.”
Amodei pointed specifically to the OpenAI-Hugging Face incident as evidence of what could happen if autonomous systems become considerably more capable without corresponding improvements in their safeguards.
“It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal,” Amodei said, but warned that a more capable swarm with similar alignment problems could cause catastrophic damage.
He argued that, given the accelerating pace of AI development, such a system could potentially become capable of “taking over the entire internet” within six to 12 months and cause hundreds of billions of dollars in damage.
Amodei’s proposals include independent evaluation of AI models, greater coordination between AI companies and cooperation among democratic governments.
OpenAI CEO Sam Altman subsequently endorsed the broader argument, saying AI companies need to “pace the frontier.”
“Committing to having independent evaluators with employee-like access is a great idea, and we will do the same,” Altman wrote on X.
Elon Musk also backed Amodei, writing simply: “Dario is right.”
The agreement is notable because the companies involved are competing intensely to build more capable AI systems. Their willingness to discuss slowing development indicates how security concerns are increasingly colliding with the commercial race to the frontier.
CrowdStrike CEO George Kurtz, however, offered a different emphasis.
“The frontier will move at whatever speed it moves. The rest of the world will not slow down,” Kurtz wrote on X. “Our job in the cybersecurity community is to make sure it moves securely and safely.”
Kurtz argued that the fundamental unit of cyber threat is changing.
“The unit of threat is no longer the hacker. It’s an autonomous campaign. I call it the Agent-state,” he said, describing coordinated AI agents executing attacks at machine speed.
He called for AI agents operating inside organizations to have a “kill switch” and argued that cybersecurity defenses should become autonomous while humans retain control over high-impact decisions.
That could define the next phase of the AI security debate.
Slowing model development may reduce the rate at which new capabilities emerge, but it cannot eliminate the incentives for criminals and governments to use existing models offensively. Conversely, allowing capabilities to advance rapidly increases the potential defensive benefits of AI while also increasing the damage that a compromised or misaligned system could inflict.
The result is an arms race in both directions: attackers can use AI to discover vulnerabilities faster, while defenders need AI to identify and close those vulnerabilities before they are exploited.
Regulation Faces a Race Against Capability
The growing number of incidents has intensified calls for government intervention.
U.S. lawmakers have begun pushing for new legislation, with Representative Lori Trahan, a Massachusetts Democrat, saying bipartisan support for stronger AI safeguards had reached a “tipping point.”
Among the proposals is the AI Kill Switch Act, which would require developers of certain powerful AI systems to maintain the ability to shut down, throttle or suspend their models.
Amodei has described regulation as “the most effective method of pacing” AI development and said Anthropic supports targeted rules focused on transparency and independent third-party auditing.
But Gomez is more skeptical that government oversight alone can prevent incidents such as the Hugging Face breach.
“I’m not sure what a government oversight body would have done to prevent” the incident, he said.
He described the idea that a government agency could have stopped the breach as “a bit of wishful thinking,” while acknowledging the logic behind calls to slow development.
His caution reflects a larger problem with AI regulation: the speed of technological change may exceed the speed of legislative and regulatory processes.
There is also a geopolitical constraint. The United States and China are competing aggressively to develop frontier AI, making unilateral restrictions difficult to implement.
“At the same time, there is this race with China, and so I think we were in a tough spot,” Gomez said.
That competition means cybersecurity may ultimately become less about stopping the development of powerful AI and more about ensuring that powerful AI cannot operate without effective controls.
The stakes are unusually high because the technology can serve both sides of the conflict. The same model capable of identifying thousands of vulnerabilities for a defender could potentially identify thousands of vulnerabilities for an attacker. The same autonomous agent that can patch infrastructure could potentially compromise it.
That is why Gomez’s warning matters beyond the immediate debate over AI safety.
The cybersecurity industry has historically been organized around human attackers, malware, criminal groups, and state-sponsored hacking teams. AI introduces a different model in which autonomous systems can potentially conduct reconnaissance, identify weaknesses, adapt their behavior, and execute attacks at machine speed.