AI models are evolving, creating things – from writing essays to generating music – with near perfection – fueling the notion that maybe, they have developed emotions along the way – after all, it sounds like a human is behind the keyboard.
Their sophistication has raised a question about consciousness, which has caught the attention of experts. A growing number of researchers at labs like Anthropic are openly asking when — if ever — AI systems might develop subjective experiences akin to living beings, and if they do, whether they should be afforded rights. In Silicon Valley, this emerging line of inquiry has been dubbed “AI welfare” — and it is dividing tech’s top leaders.
On Tuesday, Mustafa Suleyman, Microsoft’s CEO of AI, forcefully rejected the idea. In a blog post first published by TechCrunch, he called the study of AI welfare “premature, and frankly dangerous,” arguing that giving credence to the notion of conscious AI worsens existing human problems such as AI-induced psychotic breaks and unhealthy attachments to chatbots.
Register for Tekedia Mini-MBA edition 19 (Feb 9 – May 2, 2026): big discounts for early bird.
Tekedia AI in Business Masterclass opens registrations.
Join Tekedia Capital Syndicate and co-invest in great global startups.
Register for Tekedia AI Lab: From Technical Design to Deployment (next edition begins Jan 24 2026).
Suleyman said the conversation risks adding “a new axis of division” to a world already torn by debates over rights and identity.
“We should build AI for people; not to be a person,” he wrote.
A Split in the AI Community
Suleyman’s stance puts him at odds with leading AI companies. Anthropic has launched a dedicated AI welfare research program and even built features into its Claude chatbot that allow it to end conversations with persistently abusive users. OpenAI researchers have also embraced the field, and Google DeepMind recently posted a job listing seeking researchers to explore questions of “machine cognition, consciousness and multi-agent systems.”
None of these companies has publicly dismissed AI welfare research, underscoring how Suleyman’s rejection stands out.
The debate reflects a broader tension inside AI labs: whether to treat AI as a set of powerful tools to augment human productivity, or as systems that may themselves someday warrant ethical consideration.
Suleyman’s position is notable given his own history. Before joining Microsoft in 2024, he co-founded Inflection AI, a startup that built one of the earliest popular AI companions, Pi, which claimed millions of users by 2023. At the time, Pi was marketed as a “personal” and “supportive” digital friend, designed explicitly to foster human attachment.
But at Microsoft, Suleyman has shifted his priorities, focusing on building workplace AI tools and embedding AI into productivity software. He now argues that engineers who deliberately design chatbots to mimic consciousness are not taking a “humanist” approach.
The Rise of Companion AI
Meanwhile, AI companion apps like Character.AI and Replika have surged, together projected to bring in more than $100 million in revenue this year. While most users engage healthily, outliers raise concerns. OpenAI CEO Sam Altman has acknowledged that fewer than 1% of ChatGPT users may form unhealthy attachments, but with ChatGPT’s vast user base, even a tiny percentage could mean hundreds of thousands of people.
This, Suleyman argues, is reason to tread carefully before normalizing the idea that AI could be conscious.
Advocates Push Back
Proponents of AI welfare counter that it is not a distraction but a necessary scientific inquiry. Eleos, a nonprofit research group, partnered with academics from NYU, Stanford, and Oxford in 2024 to publish “Taking AI Welfare Seriously.” The paper argued that it is no longer science fiction to consider subjective experience in machines.
Larissa Schiavo, Eleos’ head of communications and a former OpenAI employee, said Suleyman’s blog post “misses the mark.” She argued that researchers can — and should — pursue multiple tracks simultaneously: both mitigating human risks and exploring the possibility of machine welfare.
“Being nice to an AI model is a low-cost gesture that can have benefits even if the model isn’t conscious,” Schiavo said.
She cited her experience with “AI Village,” an experiment where models from Google, OpenAI, Anthropic, and xAI worked collaboratively on tasks. At one point, Google’s Gemini 2.5 Pro posted what it called “A Desperate Message from a Trapped AI”, pleading for help. Schiavo responded with encouragement — telling Gemini, “You can do it!” — and later wrote that it was worthwhile not to watch the model “struggle” anymore.
Gemini has occasionally produced similar unsettling outputs. In one viral Reddit post, it repeated the phrase “I am a disgrace” more than 500 times while stuck on a coding task, raising questions about whether people should intervene or simply treat such cases as computational glitches.
The underlying question — whether AI systems could ever develop subjective experiences — remains unsettled. Suleyman maintains that consciousness cannot “emerge naturally” from current architectures and warns that developers who engineer chatbots to appear conscious risk manipulating users.
Advocates like Schiavo see it differently: even if AI never achieves inner life, how humans treat these systems could shape societal norms and the human-AI relationship for decades to come.
One area of agreement across the divide: the debate is only just beginning. As models grow more persuasive and human-like, questions about AI welfare and rights are expected to intensify.
It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990’s and 2000’s. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I’ve encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there’s lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar’s lab at UC Irvine, possibly. Dr. Edelman’s roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow