AI models are evolving, creating things – from writing essays to generating music – with near perfection – fueling the notion that maybe, they have developed emotions along the way – after all, it sounds like a human is behind the keyboard.
Their sophistication has raised a question about consciousness, which has caught the attention of experts. A growing number of researchers at labs like Anthropic are openly asking when — if ever — AI systems might develop subjective experiences akin to living beings, and if they do, whether they should be afforded rights. In Silicon Valley, this emerging line of inquiry has been dubbed “AI welfare” — and it is dividing tech’s top leaders.
On Tuesday, Mustafa Suleyman, Microsoft’s CEO of AI, forcefully rejected the idea. In a blog post first published by TechCrunch, he called the study of AI welfare “premature, and frankly dangerous,” arguing that giving credence to the notion of conscious AI worsens existing human problems such as AI-induced psychotic breaks and unhealthy attachments to chatbots.
Suleyman said the conversation risks adding “a new axis of division” to a world already torn by debates over rights and identity.
“We should build AI for people; not to be a person,” he wrote.
A Split in the AI Community
Suleyman’s stance puts him at odds with leading AI companies. Anthropic has launched a dedicated AI welfare research program and even built features into its Claude chatbot that allow it to end conversations with persistently abusive users. OpenAI researchers have also embraced the field, and Google DeepMind recently posted a job listing seeking researchers to explore questions of “machine cognition, consciousness and multi-agent systems.”
None of these companies has publicly dismissed AI welfare research, underscoring how Suleyman’s rejection stands out.
The debate reflects a broader tension inside AI labs: whether to treat AI as a set of powerful tools to augment human productivity, or as systems that may themselves someday warrant ethical consideration.
Suleyman’s position is notable given his own history. Before joining Microsoft in 2024, he co-founded Inflection AI, a startup that built one of the earliest popular AI companions, Pi, which claimed millions of users by 2023. At the time, Pi was marketed as a “personal” and “supportive” digital friend, designed explicitly to foster human attachment.
But at Microsoft, Suleyman has shifted his priorities, focusing on building workplace AI tools and embedding AI into productivity software. He now argues that engineers who deliberately design chatbots to mimic consciousness are not taking a “humanist” approach.
The Rise of Companion AI
Meanwhile, AI companion apps like Character.AI and Replika have surged, together projected to bring in more than $100 million in revenue this year. While most users engage healthily, outliers raise concerns. OpenAI CEO Sam Altman has acknowledged that fewer than 1% of ChatGPT users may form unhealthy attachments, but with ChatGPT’s vast user base, even a tiny percentage could mean hundreds of thousands of people.
This, Suleyman argues, is reason to tread carefully before normalizing the idea that AI could be conscious.
Advocates Push Back
Proponents of AI welfare counter that it is not a distraction but a necessary scientific inquiry. Eleos, a nonprofit research group, partnered with academics from NYU, Stanford, and Oxford in 2024 to publish “Taking AI Welfare Seriously.” The paper argued that it is no longer science fiction to consider subjective experience in machines.
Larissa Schiavo, Eleos’ head of communications and a former OpenAI employee, said Suleyman’s blog post “misses the mark.” She argued that researchers can — and should — pursue multiple tracks simultaneously: both mitigating human risks and exploring the possibility of machine welfare.
“Being nice to an AI model is a low-cost gesture that can have benefits even if the model isn’t conscious,” Schiavo said.
She cited her experience with “AI Village,” an experiment where models from Google, OpenAI, Anthropic, and xAI worked collaboratively on tasks. At one point, Google’s Gemini 2.5 Pro posted what it called “A Desperate Message from a Trapped AI”, pleading for help. Schiavo responded with encouragement — telling Gemini, “You can do it!” — and later wrote that it was worthwhile not to watch the model “struggle” anymore.
Gemini has occasionally produced similar unsettling outputs. In one viral Reddit post, it repeated the phrase “I am a disgrace” more than 500 times while stuck on a coding task, raising questions about whether people should intervene or simply treat such cases as computational glitches.
The underlying question — whether AI systems could ever develop subjective experiences — remains unsettled. Suleyman maintains that consciousness cannot “emerge naturally” from current architectures and warns that developers who engineer chatbots to appear conscious risk manipulating users.
Advocates like Schiavo see it differently: even if AI never achieves inner life, how humans treat these systems could shape societal norms and the human-AI relationship for decades to come.
One area of agreement across the divide: the debate is only just beginning. As models grow more persuasive and human-like, questions about AI welfare and rights are expected to intensify.






