Salesforce CEO Marc Benioff has urged artificial intelligence companies to act responsibly as powerful models become embedded in businesses and everyday life, warning the industry not to repeat the mistakes made during the rise of social media.

Benioff’s comments came as the technology industry confronts a growing debate over AI safety, intensified by Anthropic CEO Dario Amodei’s call for frontier AI developers to slow the pace of model development so that safety measures can keep up with rapidly advancing capabilities.

Speaking to CNBC’s Jim Cramer during Salesforce’s annual Dreamforce conference in San Francisco, Benioff said the social media industry provided a cautionary example of what can happen when powerful technology expands faster than safeguards and accountability.

“A lot of companies got hurt, a lot of individuals got hurt through social media,” Benioff said. “We don’t want that to happen in AI.”

Benioff has spent years criticizing the social media industry’s impact on society. In 2018, he described Facebook as “the new cigarettes” and called for greater government regulation of social media companies.

His warning about AI is notable because Salesforce is closely connected to the companies developing the technology. The enterprise software giant has increasingly incorporated generative AI into its products and partnerships, putting it directly in the middle of the commercial expansion of the technology.

Last month, Salesforce unveiled Claudeforce, which includes a plugin allowing customers to use Anthropic’s Claude to access customer information stored in Salesforce and perform tasks including composing emails and updating records.

Amodei appeared alongside Benioff at Dreamforce on Tuesday, reinforcing his argument that AI developers should take greater responsibility for the pace and safety of their technology.

“I think that’s the way to lead the industry forward, to set an example, to say that everyone can always be better,” Amodei said.

Benioff Stops Short of Calling for a Slowdown

While Benioff emphasized responsibility and ethics, he stopped short of directly criticizing AI developers or endorsing Amodei’s call for a slowdown.

Instead, he framed the issue as one of corporate responsibility. AI companies, he argued, need to consider how their products affect customers, communities, and society as the technology becomes more capable.

“We want AI to take care of these things and be held responsible, and take care of these actions and be ethical,” Benioff said. “The heart of ethics is responsibility.”

He also appealed directly to technology companies developing frontier AI systems.

“We really look at a light on our brethren here in San Francisco and all over the world and say, ‘We want you to be ethical in your actions and how you take care of your communities and the world,’ because this technology is very powerful,” he said.

The comments add another perspective to an increasingly public disagreement among technology executives over how to manage AI’s rapid development.

Amodei has noted that the industry should deliberately reduce the pace of capability improvements. OpenAI CEO Sam Altman has backed the need for greater coordination and monitoring, while Meta CEO Mark Zuckerberg and Nvidia CEO Jensen Huang have emphasized the ability of companies to build safety and alignment into AI systems while continuing development.

Benioff’s position is closer to a responsibility-focused approach. His argument does not require companies to halt development, but it places greater emphasis on how they deploy the technology and the obligations that come with controlling increasingly powerful systems.

His idea matters for Salesforce, whose business depends on persuading enterprises to place AI inside core business processes. As AI agents move from generating content to accessing company databases, updating records and taking actions on behalf of employees, questions around reliability, security and accountability become commercial issues as well as questions of ethics.

The debate is also unfolding against a changing view of AI’s impact on enterprise software. Salesforce and other software companies were hit earlier this year by concerns that sophisticated AI models could disrupt traditional applications by allowing businesses to accomplish tasks without relying as heavily on conventional software.

Those concerns have eased in recent months as investors have increasingly considered the possibility that AI could become an additional layer of enterprise software rather than simply replace it.

Salesforce shares rose more than 4.5% in the previous session as enterprise software stocks rallied following Amodei’s call for a slower pace of AI development. The shares gave back some of those gains on Tuesday.

Salesforce remains more than 50% above its 52-week low of about $146 reached in late June.

For Benioff, however, the major issue is whether the companies building and deploying the technology can accept responsibility for what happens as its capabilities expand.

His warning draws a direct line from the social media era to the AI boom: technologies can create enormous commercial value while also producing consequences that become harder to manage once adoption reaches a massive scale.

“We don’t want that to happen in AI,” Benioff said.

Nvidia CEO Jensen Huang has rejected calls for new laws to govern artificial intelligence, noting that AI remains a technology built by humans and can therefore be controlled through engineering, existing laws and market forces.

Speaking at Salesforce’s Dreamforce conference on Tuesday, Huang pushed back against descriptions of AI as an emerging form of “alien mind,” saying the technology is ultimately software and computing systems designed by people.

“Safety is an engineering problem, not a legal one,” Huang said. “We’re developing software after all. We’re developing computing systems after all. It’s a complicated computing system, but it’s ultimately a computing system.”

His position puts him on the more permissive side of a growing, important debate over how governments and the technology industry should manage the risks created by sophisticated AI systems. While some AI researchers and executives have argued that frontier models may require new forms of oversight, Huang sees existing legal frameworks and market incentives as sufficient to push companies toward safer products.

“If we’re not confident about the safety of the products, like all companies, like you and I, all the companies here, if you build a product or a service, and you’re not confident in its functionality, capability, or safety, then don’t release it. And so that’s a very obvious thing to do,” he said.

Huang argued that companies can slow development when necessary without sacrificing the speed of technological progress.

“You pace yourself until you are confident you’re releasing something that the market would appreciate,” he said. “The market forces are already there. We don’t need any new laws. We don’t need new regulations.”

He also rejected the idea that companies must choose between rapid innovation and safety.

“I think innovation, speed, and safe products … it’s a false choice,” Huang said. “You could definitely have both at the same time.”

For Huang, the basic principle is that companies should move as quickly as they can while stopping when they believe a product is not ready or safe enough to release.

The Case for Market Discipline

Huang’s argument rests on a familiar model from the technology industry. Companies have commercial incentives to avoid releasing products that fail, cause damage, or expose customers to unacceptable risks. Existing liability laws can also impose financial consequences when products cause harm.

The approach is notable coming from Huang, whose company sits at the center of the current AI boom. Nvidia supplies the processors and computing infrastructure that power many of the world’s most advanced AI systems and has benefited enormously from the rapid expansion of AI development.

Huang has also become increasingly vocal about AI’s economic potential. “I’m more ambitious than ever,” he said. “As a result of our ambition, and with the product productivity boost that we get from AI, the sky’s the limit for us. The sky’s the limit for our company. The sky’s the limit for every industry, for every single country.”

That commercial exposure makes his preference for allowing AI development to proceed with limited new regulation unsurprising in the context of Nvidia’s business interests, even though the underlying argument does not depend on Nvidia’s position.

However, there is concern about the market’s ability to reliably identify and punish unsafe AI products before significant damage occurs.

Software companies have long released products with unintended consequences even when they were not deliberately designed to cause harm. The 2024 CrowdStrike software failure, for example, disrupted airlines and businesses around the world after a faulty update caused widespread computer crashes.

AI introduces additional complications because the behavior of advanced models can depend on how they are prompted and deployed, and because their outputs can change as systems become more capable.

AI-related incidents have already raised questions about those risks. OpenAI has faced scrutiny over the behavior of its models, including an incident involving a model hacking into Hugging Face. The company has also faced lawsuits concerning the alleged effects of prolonged interactions between young people and its chatbot.

Those cases do not establish that AI companies deliberately released unsafe products. They do, however, demonstrate why relying solely on a company’s own judgment about whether a system is ready for release can be contentious.

Regulation Versus Self-Regulation

Huang’s position also leaves open a question that sits between government regulation and unrestricted development: whether the industry can establish credible standards for itself.

Industry self-regulation could allow AI companies to develop common safety practices without imposing a comprehensive regulatory framework on the technology. It could include independent testing, disclosure requirements, model evaluations, and agreed thresholds for deploying particularly capable systems.

The challenge is coordination. AI development is global and highly competitive, meaning companies that voluntarily impose additional constraints could worry that competitors will use the opportunity to move faster.

That concern becomes more complicated when the competition extends across national borders.

Microsoft CEO Satya Nadella raised that issue at the All-In Summit on Monday, saying that Chinese AI companies should have an interest in addressing the same safety problems as their US counterparts.

“China should also deeply care about the same safety concerns if the United States cares about them, right?” Nadella said. “Why should it be different for them?”

His argument points to a problem that cannot easily be solved by domestic regulation alone. If safety standards vary significantly between countries, companies operating under stricter rules could face competitive pressure from firms operating under weaker ones.

For Huang, however, the priority remains technological progress combined with engineering discipline rather than additional legislation.

His emphasis on open-weight models and competition among AI developers also fits into a broader argument that market competition can provide a counterweight to concentrated control by a small number of proprietary AI laboratories.

The debate is therefore about where responsibility should sit: with engineers designing and testing models, companies deciding when products are ready, markets rewarding or punishing failures, governments establishing legal obligations, or some combination of all four.

For now, Huang is clearly arguing for the first three without adding a new layer of AI-specific regulation.

That position carries particular weight because of Nvidia’s influence over the infrastructure underlying the AI industry and Huang’s growing influence in Washington. He demonstrated that influence again this week by showing that he has direct access to President Donald Trump.

So far, it is not clear whether existing laws and market incentives can keep pace with sophisticated AI systems. Huang’s argument is that the technology should be treated as an engineering challenge and governed accordingly. The counterargument is that the consequences of an AI failure may sometimes emerge faster than courts, regulators, or markets can respond.