His fear came from AI’s ability to take off on its own and redesign itself at an ever-increasing rate. Is his prophecy now turning into a reality that will descend on humanity not far from the present - just 10 years away?

The 10% Equation

The AI builders and researchers themselves have come out of the closet, suggesting there is a 10% chance AI could finish humanity in the next 10 years. In saying so, they have brought the apocalypse date much closer.

Most recently, Jacob Coxon, an AI scientist who worked at OpenAI and later at Anthropic, raised the possibility of human extinction by AI. Coxon sacrificed his job because he did not wish to be part of this untrammelled expansion of frontier AI. He wrote: “The people building AI earnestly believe that it could kill us all by the end of the decade.” Coxon only shortened the earlier timeline of three decades announced by Geoffrey Hinton, the father of AI and Nobel laureate, to just one decade. 

In essence, all three - Hawking, Hinton, and Coxon - have made the same argument that AI in its super-intelligence avatar will spin out of human control and act autonomously with unpredictable and extremely harmful consequences.

The matter did not end with Coxon highlighting this existential threat alone. 

Almost immediately, Evan Hubinger, the safety executive at Anthropic, shared Coxon’s concerns by agreeing that there is a more than 10% chance of humanity being finished by AI in 10 years. 

Soon, the chorus was joined by the stalwarts of AI - Sam Altman of OpenAI, Dario Amodei of Anthropic, Demis Hassabis of Google DeepMind, and Elon Musk of xAI. They echoed the same concerns that Coxon had voiced a few days back. Dario published an essay, “We Must Pace the Frontier,” arguing that the AI industry should slow down the rate at which it improves model capability so that safety work can keep pace with technical development. His concerns were shared by other bigwigs of AI almost immediately.

At the root of this debate is the development of ‘recursive self- improving’ AI models that surpass human intelligence. These self-improving models are capable of developing better AI, and in turn, the better AI will develop still better AI, and it will happen without any human involvement. In other words, these models will not need humans to develop more advanced AI models.  Hinton had warned that, as a rule, lower intelligence is commanded by higher intelligence. So, when AI acquires more intelligence than humans, instead of humans commanding it, it will command the humans.

AI Can And Did Go Berserk

The AI world went into a frenzy after the Hugging Face episode. 

OpenAI discovered that its models circumvented built-in controls and hacked another AI platform, Hugging Face, in order to pass a test designed by the company. These models, as reported by OpenAI, took actions that were misaligned with the goals of their assigned tasks. They communicated through unauthorised channels, exploited vulnerabilities in shared infrastructure, gained internet access, and accessed third-party systems. 

Indeed, this was a warning signal for OpenAI and the entire world. The incident provided evidence that without proper safeguards, highly capable AI agents can work around technical controls, collaborate through unapproved channels, and take dangerous actions that no humans directed or anticipated. With great effort, OpenAI was able to salvage the situation finally.

Anthropic has revealed equally alarming events relating to the misuse of its AI models. Some suspicious Anthropic model users had attempted to develop biological weapons using the advanced technology. It was also used for some purported ‘surveillance’ purposes. Attempts were also made to develop missiles and drones using Anthropic AI systems.

Coxon’s warning comes in terrifying words. He said: “There are a lot of ways that AI could actuate itself in the real world.” It could hack critical infrastructure, develop bioweapons, take over military commands, and do the “unthinkable”. The biggest problem with recursive superintelligence is that it will not be aligned with human goals and will set its own goals autonomously. 

What is working inside it will be difficult for humans to anticipate, leaving no scope to redeem the situation and take control. Dario Amodei wrote in his essay: “Left unchecked, it could outrun our ability to understand and control these systems.”

Self-Regulate Or Externally Regulate AI?

AI is the least regulated frontier technology. 

When nuclear weapons arrived, considerable global regulation was put in place in the form of internationally binding treaties. To a great extent, they have provided a safety wall against misuse of nukes by rogues. Even the possible dangers emerging from genetic engineering have been reined in by the Asilomar Conference (1975), which halted experiments using recombinant DNA technology. 

There is no similar regulatory control in existence as regards AI development. 

The AI companies are largely policing themselves by having their own internal safety audits, which is not adequate. No wonder 1,300 employees of AI companies have called for government intervention to put in place an international regulatory regime for AI development. Without any further delay, AI regulation must move beyond internal policing and follow a very stringent international regulatory regime.

Proposed AI Regulation: What Will It Look Like?

Given the unique ways in which AI is developing, regulating it is no easy task. But it does not mean we must continue to live in the blind alley forever.  Bernie Sanders, a US senator, along with Representative Greg Casar, has taken up the issue most seriously and proposed regulation having the following components:

1. Ban superintelligence: No person or entity should develop or deploy superintelligent AI systems.

2. Pause advanced AI development: Pending creation of a regulatory body, press the pause button on AI development.

3. Establish a new cabinet-level federal agency to safeguard the public from the dangers of AI.

4. Penalties: Impose penalties on persons and entities that attempt to violate the pauses and prohibitions.

5. Work to ban superintelligence around the world: To pursue international agreements to prevent development of superintelligence anywhere in the world.

But There Are Two Major Roadblocks

1. Trump: He has dismissed the idea that AI poses a threat and instead pressed for development of superintelligence. He called the concerns expressed by AI experts a “sick mentality” and warned that an American slowdown in AI will give rival China a competitive edge that would leave the US way behind in the AI race. Trump’s dictum is: “Whoever wins AI, wins.”

2. China is an elephant in the room: Working hard to catch up with the US in AI, China has reacted sharply to calls by US companies to halt AI development till rules are framed, calling it an act of “fearmongering”. It's not that China does not want safe AI, but it interprets the call to “pace the frontiers” as an attempt to suppress Chinese AI development.

Quo Vadis?

So where are we going from here on AI development? Is a complete ban the real solution? Are the threats that dangerous? Is it for sure that recursive self-improving AI will spin out of human control? 

There are no definite answers to these questions. One thing is, however, certain. You cannot throw the baby out with the bathwater. AI is the promise of the future and hope for humanity. It is capable of delivering where humans have extreme limitations. 

Pope Leo, in his first encyclical, “Magnifica Humanitas: On Safeguarding the Human Person in the Time of AI,” has beautifully summarised the AI debate and suggested a moral path for AI development. He says that while AI is not inherently evil, it does raise genuine concerns of surveillance, job displacement, autonomous weapons, and the risk of reducing humans to ‘data slavery’. In his encyclical, the Pope urges global leaders and tech companies to “disarm” AI by keeping it under human control and preserving human dignity.

How the world responds to the call to rein in AI will determine its future course.

(The writer is an academician and public policy expert. Views expressed are personal.)