Nobel Prize-winning computer scientist Geoffrey Hinton is calling for a major change in how artificial intelligence companies bring new systems to market, arguing that developers should have to demonstrate that their products are safe before releasing them to the public.
Hinton, widely known as one of the pioneers of modern AI, made the proposal on Tuesday during the Smart Girl Dumb Questions podcast hosted by Nayeema Raza. He compared the development of capable AI systems with the pharmaceutical industry, where companies must demonstrate the safety and efficacy of new drugs before they can be widely marketed.
“You’re not allowed to just make a new drug and release it on the market,” Hinton told Raza. “You have to convince the FDA. And to do that, you have to do a lot of work, about a billion dollars worth of work.”
Register for the next Tekedia Mini-MBA.
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
“That seems like the very least we should have for AI,” he added.
Hinton’s proposal would represent a significant expansion of government oversight of AI development. Instead of allowing companies to release sophisticated AI models and address safety problems afterward, developers would have to satisfy an independent regulator before putting certain systems into public use.
The push comes as the AI industry faces growing questions over whether existing safeguards are keeping pace with the rapid development of autonomous systems.
Hinton’s argument is rooted in the possibility that AI development itself could accelerate as companies increasingly use AI systems to improve software, research and the models that underpin the technology.
“We’re beginning to get recursive self-improvement because AI is being used to make AI better in many different ways,” Hinton said.
That prospect is integral to the safety debate because improvements in AI capabilities could become faster if models begin contributing more directly to the development of their successors. A system that helps researchers write code, conduct experiments or optimize training processes could shorten development cycles and increase the pace at which new capabilities emerge.
“Exponentials grow very fast,” Hinton said.
Hinton’s comments come amid renewed concern over the reliability and control of advanced AI systems. He pointed to OpenAI’s decision to shelve its latest model after testing raised concerns about whether the system would reliably follow users’ instructions and remain within the permissions granted to it.
OpenAI’s head of safety systems, Saachi Jain, told Business Insider that GPT-6.1 Astra had improved its ability to persist through difficult tasks but continued to fall short in areas including permissions and communicating what it had done.
Those weaknesses have become of serious concern as AI systems move beyond generating text and images toward carrying out multistep tasks on behalf of users. An AI system that can act autonomously has a different risk profile from one that simply responds to prompts because mistakes can translate into actions rather than just incorrect answers.
Hinton’s proposal effectively applies the precautionary logic used in other high-risk industries to AI. Pharmaceutical companies must produce extensive evidence before new drugs are approved because the consequences of releasing an unsafe product can extend well beyond the developer. Hinton argues that increasingly capable AI systems should face a comparable requirement before deployment.
His proposal also raises a fundamental question for regulators: what level of capability should trigger mandatory pre-release approval?
AI systems range from relatively simple applications to highly capable models that can write software, conduct research, operate tools, and potentially perform autonomous tasks. Applying the same regulatory standard to every AI product could be impractical, while setting the threshold too high could leave significant risks outside the regulatory framework.
Hinton did not specify what agency should regulate AI or what tests would be required before a model could be released. His comparison with the US Food and Drug Administration, however, suggests a system in which developers would have to provide substantial evidence that an AI product meets predefined safety requirements.
‘A Year or Two’ Before Risks Worsen
Hinton also offered a stark assessment of the speed at which AI risks could increase.
“So, it’s just a guess,” he said. “But I guess we have a year or two before everything gets much worse than it is now.”
The timeframe is explicitly his estimate rather than a forecast based on a stated model or measurable threshold. But it reveals why Hinton has pushing for stronger safeguards around advanced AI. His concern is not just that AI systems will become more powerful. It is that the mechanisms driving their improvement could themselves become more automated.
Other industry experts have acknowledged the concern is valid for regulation. Traditional product-safety regimes generally operate around a relatively stable development process in which manufacturers can test a product before it reaches consumers. If AI development accelerates through AI-assisted research and engineering, regulators could face a moving target in which capabilities change faster than existing testing and approval systems.
Hinton’s proposed model would shift the burden toward developers, requiring them to establish safety before deployment rather than relying primarily on post-release monitoring.
Such a system could also increase the cost and time required to develop advanced AI. Hinton’s comparison with drug development is deliberate: he cited the roughly billion-dollar scale of work involved in convincing the FDA that a new drug is safe, suggesting that rigorous AI testing may need to become a major cost of developing frontier systems.
But that could alter competition in the industry. Large AI companies may be better positioned to absorb extensive testing and regulatory costs, while smaller developers could face greater barriers to releasing advanced systems.
At the same time, a formal approval system could give users and businesses greater assurance that highly capable AI products have undergone independent safety evaluation.
The debate is becoming more consequential as AI companies push toward systems capable of acting with greater autonomy. Hinton’s central argument is that the industry should not wait for a major failure before imposing stronger safeguards.
His comparison with pharmaceuticals captures the broader shift in the AI safety debate: there is now a growing belief that if advanced AI systems can increasingly act in the real world and potentially contribute to their own improvement, safety testing may need to become a prerequisite for deployment rather than an exercise conducted after a product reaches users.