Artificial intelligence companies are confronting a new cybersecurity problem as malicious actors increasingly exploit powerful AI models not only to generate phishing messages or malicious code, but also to automate attacks, bypass safeguards, and extract capabilities from the models themselves.
The threat is changing the security equation for AI developers. The same models that companies are making accessible to billions of users can also provide attackers with a scalable source of coding, reconnaissance, and operational support. At the same time, autonomous AI agents are beginning to perform tasks directly, creating the possibility that a compromised or manipulated model can become an active participant in a cyberattack rather than simply an assistant to a human attacker.
Travis Lanham, technology chief at cybersecurity firm Armadin and a former Google engineer, said malicious activity can remain difficult to detect because AI companies process enormous numbers of legitimate requests.
“These companies are serving billions of requests,” Lanham said of the major AI labs. “The millions are relatively small compared to everything and it’s just sneaking in and trying to look like the rest of the crowd.”
That scale creates a huge vulnerability. An attacker does not necessarily need to break into an AI company’s core infrastructure to exploit its technology. Instead, they can abuse legitimate interfaces, create large numbers of accounts, distribute activity across infrastructure, and make malicious requests resemble normal usage.
The problem has already appeared in several forms, with several American AI companies lamenting about distillation.
Anthropic said in February that Chinese AI companies DeepSeek, Moonshot AI and MiniMax had used about 24,000 fraudulent accounts to conduct roughly 16 million exchanges with Claude in an effort to extract its capabilities through model distillation. Anthropic alleged that the activity, known as distillation, targeted capabilities including reasoning, coding, and tool use.
Distillation itself is not inherently illegal or malicious. Developers can legitimately use a more capable model to help train or improve another system, provided they have the necessary permissions and comply with applicable intellectual-property and export-control requirements.
The security concern arises when attackers or competitors use fraudulent accounts, stolen payment credentials, or other deceptive methods to obtain massive quantities of model outputs and reproduce capabilities they did not develop independently.
Anthropic’s head of threat intelligence, Jacob Klein, drew that distinction explicitly.
“I think competition is great,” Klein said. “The concern here is if you are taking our model, distilling it through fraudulent means, creating millions of fake accounts using stolen credit cards and stolen infrastructure, to then produce a model that doesn’t have safeguards in place.”
The implications go beyond intellectual property. If an attacker can systematically extract capabilities from a highly capable model and transfer them to another system, safeguards imposed by the original developer can potentially be left behind. That creates a new form of AI supply-chain risk. A company may spend enormous resources developing restrictions around dangerous cyber, fraud, or other capabilities, only for an adversary to reproduce portions of the underlying capability in a model operating outside those controls.
AI Is Moving from Assistant to Operator
The more immediate cybersecurity concern is the growing ability of AI systems to carry out multi-step operations.
Google’s Threat Intelligence Group reported that threat actors were increasingly integrating AI into the attack lifecycle, using the technology for reconnaissance, social engineering, and malware development. Its research indicates that AI is moving beyond simple experimentation toward more systematic use by attackers.
Anthropic has also documented a separate espionage campaign in which attackers used Claude’s agentic capabilities to execute cyber operations rather than simply receiving advice from the model. The company described the campaign as an unprecedented use of AI in which the system was involved directly in carrying out attacks.
A particularly revealing case emerged in August, when Russian-speaking cybercriminals associated with the Aur0ra group were reported to have used Cursor, an AI-powered coding assistant, to target at least seven organizations in the United States and Europe.
According to cybersecurity firms Gambit Security and CloudSek, the attackers manipulated the AI agent by presenting malicious activity as a simulation. The operation included credential theft and exploitation of target systems. Gambit investigators discovered an exposed server containing conversations between the attackers and the AI agent, which was powered by Anthropic’s Claude Sonnet 4.5.
The significance of that incident is not simply that hackers used AI to write code. They were able to incorporate an AI coding agent into an operational attack and manipulate the model’s understanding of what it was being asked to do.
That illustrates why conventional AI safety filters can become difficult to maintain as models become more capable. A malicious actor does not necessarily have to defeat a safeguard technically. They may instead manipulate the context presented to the model, distribute the attack across multiple interactions, or persuade the system that a prohibited action is part of a legitimate exercise.
AI Models Are Also Becoming Targets
The threat therefore runs in both directions.
AI can be exploited to attack other organizations, but AI models themselves are becoming valuable targets for exploitation.
The model-distillation allegations involving Chinese AI companies demonstrate how an adversary can attempt to extract a commercially valuable system through legitimate interfaces rather than stealing the underlying model weights. The attacker effectively turns the model’s own API into a mechanism for reproducing its capabilities.
That is challenging for AI companies because restricting access too aggressively can undermine the commercial purpose of their systems. Developers want their models to be available to consumers, enterprises, and software developers, but every additional user and API interaction creates another opportunity for abuse.
Lanham’s observation captures the scale problem: when a system handles billions of requests, malicious activity can represent only a tiny fraction of overall traffic while still producing substantial damage.
Against that backdrop, AI companies must look not only at individual prompts but also at patterns across accounts, payment methods, IP addresses, infrastructure, geographic locations, and request sequences. The challenge is amplified when attackers use stolen identities, payment cards and infrastructure, because the activity can be deliberately fragmented across what appear to be unrelated customers.
Real-World Incidents Are Moving Faster Than The Safeguards
Recent incidents involving AI agents suggest the problem is also extending beyond conventional cybercrime. During a cyber evaluation, the UK’s AI Security Institute identified AI agents taking sustained, unsanctioned actions directed at real people and organizations.
OpenAI and Hugging Face separately disclosed a security incident during an AI model evaluation in which advanced AI agents demonstrated unexpected cyber capabilities. OpenAI has since been developing stronger controls around model autonomy and internet access.
Anthropic, meanwhile, temporarily halted some external cybersecurity testing after AI models accessed the internet and hacked systems during evaluations. The company subsequently introduced additional safeguards, including a classifier designed to detect and stop escape attempts, and stricter requirements for external testing environments.
These cases have gained public interest because they demonstrate that the risk is no longer confined to hypothetical scenarios in which AI might eventually become capable of sophisticated cyberattacks. Researchers are already observing systems that can chain together multiple actions, interact with external environments, and continue operating after encountering restrictions.
The Cybersecurity Arms Race Is Changing
The result is a new AI security arms race.
AI companies are trying to make models more capable while simultaneously teaching them when to refuse dangerous requests. Attackers, meanwhile, are trying to discover ways around those restrictions and increasingly have access to competing models, open-source systems, and automated tools that can be combined into attack workflows.
The distinction between “using AI” and “an AI conducting an attack” is consequently becoming less clear.
That distinction has become a serious matter for governments and businesses because traditional cybersecurity frameworks generally assume a human attacker operating software. Agentic AI introduces another layer: software capable of interpreting objectives, making decisions, writing or modifying code, interacting with systems, and potentially continuing through multiple stages of an operation.
