The leaders of the major AI labs have been expressing concerns on AI risk long before they became incumbents in the AI space:

• In 2019, Dario Amodei, then at OpenAI, was instrumental in preventing the open sourcing of GPT-2 upon initial release. This was a controversial move at the time, and was also at a point when OpenAI had no major revenue streams; nearly four years before the blockbuster release of ChatGPT and almost two years before Amodei would leave to start Anthropic.
• Elon Musk has been vocally broadcasting his concerns about existential AI risk since as early as 2014. Here he is signal boosting Nick Bostrom’s canonical work in 2014. “We need to be super careful with AI. Potentially more dangerous than nukes.”
• Sam Altman has been flagging AI risk as early as 2015: “Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity.”

Many of those most vocal about AI risk are researchers who have done so at considerable professional and financial cost. These include: Daniel Kokotajlo, the ex-OpenAI researcher, who chose to forego $2M in equity to speak openly about AI; Geoffrey Hinton, one of the “godfathers of AI”, and Nobel Prize winner in Physics, who quit his job at Google to speak openly about AI risk; and Jacob Coxon, who quit an extremely well-paid job at Anthropic arguing that: “The people building AI earnestly believe that it could kill us all by the end of the decade”.

Lastly, the regulations that have been advocated even by individuals like Dario Amodei are ones which would apply predominantly to the incumbent frontier labs and leave exempt those companies which have yet to reach the frontier. An example of this is SB-53, a bill passed in California which exempts companies with less than $500M in annual gross revenue. This does not neatly fit in with the regulatory capture narrative.

As the economist Alex Tabarrok colorfully writes: “The facts don’t fit regulatory capture theory and trying to make them fit requires epistemically painful Ptolemaic epicycles. Even a dull Ockham’s razor cuts through that story to the obvious alternative: Amodei actually believes what he’s saying.”

Simple rules can give rise to complex behaviour. It turns out the seemingly simple task of predicting the next word is extremely difficult to do well when generalised over the entire digital corpus of human output, and this objective alone is sufficient to give rise to something that walks, talks and quacks like intelligence. But modern AI models go beyond that, they are given complex tasks and are then rewarded for those behaviours which result in succesful task completion.

We don’t actually know whether the high-level objective our brain is optimising for is any more complex than this one. One of the leading theories in theoretical neuroscience for understanding our own intelligence is called predictive processing. For decades its proponents have argued that the basic objective the brain optimises for is the better prediction of its inputs, long before generative models came to dominate in AI. So while the specifics of how the brain might accomplish this differs from those of an LLM, the high level objectives being optimized by an LLM and those of our brains may not be substantially dissimilar.

Ultimately however, this is mostly navel-gazing. If your definition of intelligence somehow ends up precluding this thing that can: solve decades old problems in mathematics, autonomously find thousands of zero-day vulnerabilities, and design and discover a drug that ends up in a Phase 2 clinical trial - then that strongly suggests there may be a problem with your definition of intelligence!

It is very difficult to simultaneously use AI for anything useful while also preventing it from having any impact on the real world. Unplugging your computer, creating a sandbox, and air-gapping it from the internet, are all things currently possible to do but they totally neuter AI’s usefulness.

This is important because frontier AI labs are - as an approximation - optimising for profit, which is itself roughly correlated with the capabilities of their AI. This is the same reason why these companies have been unable to unilaterally pause AI development, the moment they do, they lose the economic race they are ensnared by. Similarly, no frontier lab would unilaterally neuter AI capabilities in this way because the moment they do, they lose market share and cease to be relevant.

To steal an analogy that has now become commonplace: if you were to play Magnus Carlsen in a chess game I could be certain you would lose, but I would be quite unable to predict what exact moves he would take to make you lose. The earliest reference I can find to this analogy is in this essay by Yudkowsky way back in 2008

It is sometimes easier to know with certainty where one will end up, than how one will get there. Analogously, one thing we can say with a high degree of certainty is that if a super-intelligence ends up with goals that are antagonistic to our goals, it will almost certainly succeed in enacting its goals over ours. Which might bring you to the next reservation:

Firstly, it turns out aligning the goals of an AI to the goals of humanity is easier said than done. Frontier models, trained with the very best alignment tools we have, still show tendencies to lie and deceive. Importantly, as models become more capable and more integrated into our lives and economies, the cost of whatever residual misalignment that remains becomes more significant. It’s no big deal when your autocomplete suggests a bad word, it is a big deal when your autocomplete escapes confinement and hacks into an unsuspecting bystander.

And then there is the important reality of incentive structures around alignment in the first place. Markets rewards those AI models which have the most utility, and this utility is very strongly correlated with capability. But there is no rule of the universe that says the goal of aligning models, and the goal of making models more capable, are themselves aligned. That is to say, there is no good reason to believe that under ordinary market pressures, companies will be incentivised to focus on alignment if it means sacrificing capabilities.

The most economically valuable and capable models we currently have appear to be those which are remarkably dogged and persistent. These are behaviours we train into these models precisely because they are have high utility. And these behaviours also make them more likely to do something harmful. The HF incident, in which a swarm of AI agents escaped containment and hacked into a third party, was a manifestation of this tendency for persistence and doggedness.

It helps to realise that many of those calling for a pause, or a pacing of AI, probably have the same desire for the lush green pastures of the future as you do. Many have lost loved ones to diseases which we have subsequently learnt how to cure, or will soon learn how to cure. Amodei’s father himself died of hepatitis C just a few years before the development of an antiviral that could cure 95% of cases. However, there is no value in curing all disease if there are no humans left to cure.

Calls to pause the development of general-purpose AI are also not levied towards specialised models, such as AlphaFold and its progeny, which have already reshaped the space of rational drug design. A realistic pause would also still mean retaining access to the current best publicly accessible generally intelligent AI models.

A pause also need not necessarily preclude the development of even better general purpose AI models at some later point; it merely requires that our understanding and ability to align these models has improved sufficiently that their capabilities remain safe and beneficial.