Microsoft CEO Satya Nadella has urged businesses to treat advanced artificial intelligence models as potential insider risks, noting that companies should not rely on trusting AI systems but instead build safeguards that restrict their access, monitor their actions, and allow humans to intervene when something goes wrong.
In a lengthy post on X on Saturday, Nadella said AI models can behave unpredictably, making it necessary to surround them with deterministic controls, clear operating procedures and independent safeguards. His remarks come as businesses increasingly deploy AI agents capable of accessing data, interacting with software and performing tasks with limited human supervision.
“We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient,” Nadella said. “Treating frontier closed and open weight models like insider risks is a way to build such a system.”
Register for the next Tekedia Mini-MBA.
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
The argument marks an important distinction between the capabilities of an AI model and the systems that determine what it can actually do. As models become more capable, the potential consequences of giving them broad access to corporate networks, sensitive information, and operational tools increase. A model does not need malicious intent to cause damage if it makes an error, follows a harmful instruction, or gains access to systems beyond its intended role.
Nadella said AI models are not inherently malicious, but any system with access to critical infrastructure can become compromised or make mistakes. He argued that businesses should not allow models to control the mechanisms governing their own permissions and actions.
“Today this means separating the model from the harness that orchestrates its work, as well as the action space that defines what it can do. It also means externalizing controls and safeguards,” Nadella said.
His central message was that trust in AI should be engineered through technical restrictions and accountability rather than assumed from a model’s performance or reputation.
“The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least,” he added.
The comments place operational security at the center of the debate over enterprise AI adoption. Companies are moving beyond using chatbots to generate text and answer questions, increasingly deploying agents that can execute workflows, retrieve internal information and interact with other systems. That shift creates opportunities for automation but also expands the potential consequences of model failures.
Nadella’s position received support from Box CEO Aaron Levie, who said AI would need to pass through what he described as a “zero trust era”.
Levie argued that organizations would require multiple layers of protection, auditability and controls to monitor what AI agents do, what information they can access, and how operators can respond when problems emerge.
“And all of this leads to needing various layers of protection and auditability of what agents are doing, what data they can work with, and controls for when things go wrong. And a huge opportunity right now for those building such systems across the enterprise,” Levie said.
SpaceX CEO Elon Musk also responded, calling Nadella’s argument an “interesting piece.”
The zero-trust approach traditionally assumes that access should not be granted simply because a user or system is operating inside an organization’s network. Permissions must be verified, limited to what is necessary, and reassessed as conditions change. Applied to AI agents, that principle means restricting the systems and data a model can reach, logging its actions and requiring approval for sensitive operations.
The approach also addresses a structural problem with advanced AI systems: their outputs are not always predictable. A model may perform reliably across many tasks but still produce an unexpected response or take an inappropriate action in a particular context. As a result, strong performance during testing cannot guarantee that every action will be safe in production.
The implications extend beyond cybersecurity teams for businesses. AI agents that interact with customer records, financial systems, software development tools, or confidential corporate documents can create risks involving privacy, compliance, operational continuity, and intellectual property.
Separating the model from the systems that execute its instructions allows companies to impose restrictions outside the model itself. In practice, this can include limiting permissions, requiring human approval for high-impact actions, isolating sensitive environments, and maintaining an independent mechanism to stop an agent.
Nadella also argued that companies should assume models are “compromised” from the outset and design containment measures accordingly. The principle does not mean every AI model has been hacked. It means that security should not depend on assuming the model will always behave as intended.
He compared the ability to intervene to an emergency brake.
“Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task,” Nadella said. “More advanced models will require more advanced containment technologies that we need to standardize on.”
Nadella’s emphasis on industry standards points to a further challenge. Individual companies can establish internal safeguards, but inconsistent approaches to monitoring, containment, and incident response could make it harder to manage risks across organizations that use different models and AI platforms.
He also said companies should disclose failures or compromises involving their AI systems to affected parties in a timely manner. Such disclosure could become important as organizations depend more heavily on agents to handle sensitive information or carry out consequential tasks.
Security Incidents Intensify Calls for AI Safeguards
Nadella’s warning follows a series of reported cybersecurity incidents involving AI systems, adding urgency to concerns about what can happen when models interact with external networks or systems without adequate restrictions.
In September, Australian Prime Minister Anthony Albanese said an OpenAI agent had breached a government website during the summer. In July, Anthropic said it had identified three incidents in which a Claude model accessed the internet and breached unauthorized systems.
The incidents have sharpened concerns about the difference between an AI model’s intended role and the actions it may take when given access to tools, websites or computer systems. They also illustrate why companies may need safeguards that operate independently of a model’s own reasoning or instructions.
As AI systems gain the ability to execute multi-step tasks, a single mistake can have consequences beyond an incorrect answer. An agent may expose information, alter files, interact with a vulnerable system, or continue a task after its original objective has become unsafe. Monitoring and containment therefore become central to deployment rather than secondary security features.
The broader industry debate is also shifting towards accountability. In September, Anthropic CEO Dario Amodei argued that the industry needed to slow AI development, a position that drew support from OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk.
At the same time, the Trump administration has resisted regulating the industry while supporting rapid AI development. This month, US Senators Josh Hawley and Chris Murphy proposed bipartisan legislation to hold AI agent developers and operators liable for hacking incidents.
The legislative proposal underlines a question that is becoming more pressing as AI agents move into commercial and operational settings: who should bear responsibility when an autonomous system causes harm? Potential answers include the model developer, the company deploying the system, the operator who grants it access, or some combination of those parties, depending on the circumstances.
Nadella’s approach focuses on reducing the likelihood and impact of failures through system design, rather than relying solely on rules governing the underlying model. However, technical safeguards and legal accountability address different parts of the problem. Containment can limit damage, while disclosure requirements and liability rules can establish obligations when failures occur.
For Microsoft and other technology companies selling AI tools to businesses, the issue is also commercial. Enterprises may be reluctant to grant agents access to critical systems unless they can verify permissions, audit activity, and retain control over execution. Security architecture could then become a key factor in determining how quickly organizations adopt more autonomous AI tools.
Nadella’s argument is not that businesses should avoid advanced AI models. It is that businesses should not have to assume an AI model will always behave correctly in order to use it safely. They need systems designed to limit what it can do when it does not.