Could AI actually destroy humanity within the next five years?

A few days ago, I came across a series of posts from Jacob Coxon, an AI researcher who recently resigned from Anthropic.His warning was pretty damn alarming.According to Coxon, the major AI labs are racing toward self improving AI and eventually superintelligence. And if that process goes badly, he believes AI could potentially cause human extinction before the end of this decade.Normally, I would read something like this, roll my eyes, and go back to whatever I was doing.But then something interesting happened.His post went massively viral. And researchers and leaders from inside major AI labs began publicly expressing similar concerns.These aren’t random people on X predicting that Skynet is coming.They’re people who actually work on these systems.For example, OpenAI Chief Scientist Jakub Pachocki has called for extreme caution around the continued rapid rise of machine intelligence. Anthropic researcher Evan Hubinger has discussed a greater than 10% chance of AI killing all humans within the next decade.Now, I’m skeptical of a lot of AI doomerism.I don’t believe we’re going to wake up one morning and discover that ChatGPT has become a conscious machine god that has decided humanity is cringe and needs to be deleted.That sounds more like an AI generated cosmic horror novel.But after thinking about this for a while, I’ve come around to a much more disturbing possibility:AI doesn’t need to become superintelligent to cause catastrophic damage.And honestly, I think this is the scenario we should be paying much more attention to.First, Let’s Talk About What an LLM Actually IsA lot of the public conversation about AI starts with an assumption that today’s AI systems are basically digital versions of human brains.They’re not.A large language model is, at its core, a gigantic mathematical system trained on enormous amounts of data.You give it text, and it predicts what comes next.One token at a time.That’s why I like the phrase probabilistic parrot.Today’s LLMs can produce astonishingly convincing demonstrations of reasoning and intelligence, but that doesn’t mean they possess a human-like internal thought process.When nobody is prompting an LLM, there isn’t some little digital person sitting inside the data center thinking about life.There is no continuous internal monologue.And there’s another important limitation:The model itself doesn’t learn from your conversation in the way a human does.Once a model has been trained and released, its underlying parameters are essentially fixed until another training process creates a new model. What looks like memory is often implemented through external systems that retrieve information and feed it back into the model’s context.This distinction matters enormously when we start talking about recursive self improvement.The Myth of the AI TakeoffThe classic AI doomer scenario goes something like this:AI builds a better version of itself.That better AI builds an even better version.The next version is smarter still.The process accelerates exponentially.Eventually, the AI becomes so intelligent that humans can’t understand or control it.And then we’re screwed.This is the idea behind the so-called intelligence explosion or recursive self improvement.There is just one problem.That’s not really how frontier AI development works today.Building a frontier model is an enormous industrial process involving pretraining, training, post training, evaluation, data generation, engineering, research and many other steps.AI models are absolutely being used to help build the next generation of AI.But they’re being used as tools inside a much larger human controlled process.Researchers use AI to write code, fix bugs, conduct research, generate synthetic data, evaluate outputs and accelerate other parts of the development pipeline.And there’s something else people sometimes forget.All of this requires an enormous physical infrastructure.AI needs chips.Those chips need data centers.Data centers need electricity, cooling, water, networking, manufacturing capacity and raw materials.None of that infrastructure is currently controlled by the AI.So the idea that today’s LLM is simply going to disappear into a recursive loop and autonomously bootstrap itself into a machine god is, at least for now, highly speculative.But here’s where things get interesting.Because I think we’re focusing on the wrong problem.The Real Danger: Derivative InnovationI don’t think today’s AI needs genuine superhuman intelligence to b