Not AI doom

Somebody was bound to sound the alarm eventually. In this case it was a young British artificial intelligence researcher called Jacob Coxon, who quit his job at the AI company Anthropic to warn the world that “if we don’t slow down the current rate of progress, there is a strong chance that we could all die in the next two years.” “It’s not at all an exaggeration to say that the people who are involved with both founding these companies and building the tech believe there is a possibility of human extinction,” Coxon said. And the floodgates opened. On 8 September Coxon posted his message on as many sites as possible (and also got ten friends to retweet them), but he probably needn’t gone to all that trouble. The AI community was ready to hear that news, and four days later his former boss Dario Amodei, the CEO of Anthropic, tacitly agreed. In a 3,800-word online essay entitled ‘We Must Pace the Frontier’, Amodei made the same argument in more temperate language. “Addressing the risks requires pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of A.I. models.” He also unilaterally committed Anthropic to embed independent examiners within the company with permanent employee-level access to all its systems, so that they can verify its safety measures and report on any incidents. In the secretive and highly competitive world of the ‘frontier’ AI companies, this is unprecedented. Does Amodei mean it? It seems so, and the CEO of the other ‘frontier’ company, Sam Altman of OpenAI, immediately agreed. “We need to pace the frontier,” he wrote on X, and called the proposal for independent evaluators “a great idea”. Even Elon Musk, a dedicated contrarian and CEO of xAI (which trails the two industry leaders), said simply “Dario is right.” It’s a good start, and a whole lot better than the response to the development of nuclear weapons 80 years ago (to pick an example not exactly at random), because the potential cost of getting AI wrong could be as grave. US President Donald Trump missed the point as usual, saying on 10 September that he was concerned that “if we don’t win AI, we’re going to be put in a very bad position”, but this time the leading AI companies may well defy him. Their leaders are very bright people, and they understand how big and dangerous this moment is. The risk of an actual extinction event is probably vanishingly small. As Jim Lovelock once said to me when we were discussing possible climate disasters: “The human race is a very tough species. There will probably always be a few breeding pairs.” But most of us would prefer to avoid even much smaller catastrophes, and AI is quite capable of causing those. The trick is to keep these hugely complex computer programmes ‘aligned’. They must be ‘helpful, honest and harmless’ (the three Hs), they must do what people actually want, avoiding shortcuts and loopholes, and they must genuinely pursue the intended goal rather than faking compliance to ‘pass’. This is not just a matter of writing code. The ‘models’ must be taught these values by working through many examples. That will take the time that it takes, and going too fast lets bad behaviour slip through. Slow down, and it will probably be fine (and very useful). Go too fast, and bad things will happen. The leading companies are working in a fiercely competitive environment both at home and internationally and behaving in accord with these harsh realities requires great restraint. In such circumstances it is reassuring and even remarkable that the market leaders are actually trying to slow things down. Even an agreed ‘slowdown’ domestically will be hard to negotiate and enforce. Including China in such an agreement (it’s the main rival to the US companies, though a bit behind them) may be impossible. The whole proposal may fail, in which case we will all have to live with the consequences. But at least the key American companies are genuinely trying. And if you are still having nightmares about the machines taking over, remember Doctor Who and the cartoon of the evil Daleks looking in despair at the bottom of a flight of stairs. Daleks move on wheels, of course, and one dejected Dalek says to another: “There goes our plan for conquering the galaxy.” You can ensure humanity’s future just by ensuring that we never ‘embody’ our AI devices in robots capable of manipulating the physical universe. Disembodied intelligences can poison the water supply, make planes crash, even devise worse versions of the Black Death, but you can always pull the plug out – and they can’t plug it back in. Gwynne Dyer’s new book is ‘Intervention Earth: Life-Saving Ideas from the World’s Climate Engineers’. The previous book, ‘The Shortest History of War’, is also still available.