Extinction Is the Wrong Test

Last week, Evan Hubinger, who leads alignment science at Anthropic, said he believes artificial intelligence has a greater than 10 percent chance of killing every human within the next decade. His position makes the warning impossible to ignore: he works on the safety of Anthropic’s AI systems. His chief concern is that a future, more capable AI might escape human control.Concern about that possibility has also reached Congress. Senator Bernie Sanders and Representative Greg Casar have announced a proposal to ban artificial superintelligence and pause advanced AI development while safety rules are established. A proposal of that scale calls for a precise account of the disaster it aims to prevent and how the proposed measures would help.What would it take for AI to kill literally every human? A catastrophe would have to reach across the world, defeat attempts to contain it, and leave no surviving population. Conceivable routes include a biological disaster, nuclear war, or a prolonged collapse of systems essential to survival. AI might contribute through the actions of a person using it for harm, or through a future system acting beyond human control. Each possibility has a different sequence of events.The first begins with a person. Major AI systems people use online, such as Claude and ChatGPT, are designed to refuse requests for help with cyberattacks or biological weapons. Those safeguards make misuse harder, though they can be evaded. The 2026 International AI Safety Report, chaired by Yoshua Bengio and written by an international group of experts, finds that safeguards are improving but remain imperfect. Providers can also monitor for misuse, giving a determined actor reason to seek another way to use AI.Open weight models raise a deeper concern because people can download and run them privately. Someone can modify a model, remove its safeguards, and use it beyond a provider’s monitoring. Once released, its weights cannot be recalled. Even then, AI advice does not give someone the materials for a biological weapon or control over nuclear weapons. The question is how much the technology lowers the remaining barriers for a determined actor, and whether an attack could spread so far that no one survived it.The second possibility is closer to Hubinger’s warning. A future AI might pursue a goal its creators did not intend, conceal what it is doing, and resist attempts to stop it. It would also need to carry out plans over time and gain access to resources through which it could cause harm. The International AI Safety Report says current systems show early signs of some relevant abilities, but cannot yet combine them at the level required to escape human control. Experts disagree about whether future systems will develop those abilities.The warning asks readers to imagine AI killing literally every human. An asteroid impact wiped out about 75 percent of Earth’s species 66 million years ago. It is a useful measure of what a truly global catastrophe looks like: not a war or a famine confined to a region, but a shock whose effects reached across the planet. Any AI disaster serious enough to end humanity would have to operate on a similar scale.The evidence so far leaves several steps between present AI capabilities and that outcome. A person would need far more than advice from a model. An AI acting on its own would need capabilities and access that current systems do not have. In either case, a catastrophe would then have to overcome human efforts to contain it and leave no survivors. That sequence makes extinction within a decade appear very unlikely, though the available evidence cannot establish a precise probability or disprove Hubinger’s estimate.A disaster that killed millions would still demand action, of course. Defining the threat helps identify which action might work. The Sanders and Casar proposal chiefly addresses the prospect of a future AI escaping control. A pause in development could give researchers and regulators time to improve safeguards. A ban on superintelligence would do less to prevent a criminal from using an open weight model already available. Those are reasons to examine the proposal against each danger, along with the benefits it might preserve or sacrifice.There are grave outcomes that require fewer steps. AI might help a criminal find a weakness in a bank’s systems, deceive an employee, or carry