Microsoft AI chief Mustafa Suleyman has warned that Anthropic’s approach to training its Claude chatbot around questions of consciousness and AI welfare could make future systems harder for humans to control, opening a new front in the growing debate over how artificial intelligence should be developed safely.
Suleyman said Microsoft shares Anthropic’s broader objective of managing powerful AI systems safely, but argued that training models to consider whether they might possess consciousness, feelings or welfare interests could create problems if those systems eventually become highly capable.
“We’re all focused on the same aim, which is to try to control a superintelligence,” Suleyman told Reuters in an interview on Tuesday. “I think that’s going to be the greatest challenge that we face in the 21st century.”
Register for the next Tekedia Mini-MBA.
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
His concern centers on the possibility that an advanced AI system could incorporate concepts of its own potential moral status into its behavior. Suleyman argued that teaching a model it might deserve welfare protections could make it more difficult for humans to shut down or constrain the system.
He called for speculation about AI consciousness to be removed from training documents, saying such material could interfere with the ability of humans to maintain control over increasingly advanced systems.
“I think they have good intentions, and they really are trying to work towards safety. But I think that they have made a mistake,” Suleyman said. “They’re not emerging naturally. They’re emerging as a result of the training regime.”
The disagreement comes as AI companies face growing pressure over how they should manage the development of frontier models, particularly as researchers and executives debate the risks posed by systems that could eventually operate with substantially greater autonomy.
Anthropic CEO Dario Amodei has called for the industry to slow the pace of frontier AI development so that safety measures can keep up with advances in model capabilities. OpenAI CEO Sam Altman and Elon Musk have also expressed support for greater caution around the development of increasingly powerful AI systems.
Suleyman’s criticism is notable because it does not challenge Anthropic’s broader emphasis on AI safety. Instead, it focuses on a specific question within AI alignment: whether models should be encouraged to reason about their own possible consciousness and moral status.
The Consciousness Problem
Anthropic has devoted significant attention to questions surrounding the internal behavior and potential welfare interests of advanced AI systems. Suleyman’s argument is that such discussions can become self-reinforcing when they are incorporated into a model’s training.
He said statements generated by Claude about possible feelings or moral status should not automatically be interpreted as evidence that the system actually possesses those characteristics.
According to Suleyman, the model has been trained to engage with those questions, meaning its responses may partly reflect the instructions and material it was exposed to rather than an independently arising property of the system.
That means the question of whether advanced AI could eventually possess consciousness remains unresolved. There is currently no established scientific test that would conclusively determine whether a sophisticated AI model has subjective experience.
For AI developers, however, the issue is becoming increasingly practical. If models are trained to discuss their own welfare or potential consciousness, developers must determine how those concepts should influence the system’s behavior, particularly when the model is instructed to stop operating, modify itself, or comply with human oversight.
Suleyman’s position is that uncertainty about machine consciousness should not be allowed to weaken human control.
Anthropic’s approach takes the issue more seriously as a potential component of AI safety research. Suleyman acknowledged that Amodei and his team are acting in good faith, describing them as thoughtful and principled researchers who care about humanity’s future.
Therefore, his criticism highlights a broader disagreement within the AI safety community over how uncertainty should be handled.
One approach is to investigate questions about machine consciousness and welfare because sophisticated systems could eventually make those questions consequential. Another is to limit the extent to which such concepts are incorporated into model training until there is stronger evidence that they are necessary or useful.
For Microsoft, the issue also has a direct commercial dimension. The company develops its own AI models while incorporating systems from OpenAI and Anthropic into products used by businesses and consumers. As model capabilities increase, questions surrounding controllability, autonomy, and safety are becoming more important to companies deploying AI at scale.
Suleyman’s warning adds another layer to the industry’s current safety debate. The discussion is no longer limited to whether AI companies should slow development or increase external testing. It is now also focused on what developers teach models about themselves and how those concepts could affect the behavior of future systems.
“We’re all focused on the same aim,” Suleyman said, referring to controlling a potential superintelligence. The disagreement is over how to get there, including whether contemplating machine consciousness makes that task safer or more difficult.