The God We Deserve: Nonzero's Robert Wright on AI as Humanity's Ultimate Test (The Cognitive Revolution w/ Nathan Labenz)

(Show notes via The Cognitive Revolution)Robert Wright describes himself less as a Forrest Gump of AI history than a Zelig — a journalist who keeps turning up at pivotal scenes without ever quite being the protagonist. He interviewed Geoffrey Hinton in 1983, when neural networks were still a maverick faith and one of Hinton’s colleagues told Wright that to “hear the gospel” he’d need to talk to its evangelist. He had Eliezer Yudkowsky on his podcast in 2010, mid-transition from singularity enthusiast to doomer. His new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, is written less for the AI-pilled than for “your aunts and uncles” — readers who sense something big is happening and want to understand why it got so big so fast.The book opens with a confession of error. In a 1984 piece, Wright assumed AI would work by humans first understanding the mind and then translating that understanding into machines — the very premise of the 1956 Dartmouth conference that coined “artificial intelligence.” Deep learning inverted that. Nobody told the models that meaning is a property of words; trained only to predict the next token, they reverse-engineered cognitive functionality that evolution had built into us. Wright pushes this further with a claim he tests on Nathan: LLM training is “at least as much a process of natural selection, of evolution, as of learning” — doing millions of years of evolution in a few months. Nathan connects it to the sample-efficiency puzzle that Dwarkesh Patel keeps returning to (and to Richard Sutton‘s blank-slate Skinnerian framing), where pre-training substitutes for the hard-coding evolution gave humans.But the evolutionary lens that worries Wright most operates at a second level: selection among models in the marketplace. “Evolution asks not what traits are possible, but what traits get selected — and that question isn’t going to be decided by alignment researchers.” His unsettling argument is that the market doesn’t actually want a perfectly aligned, perfectly honest model. We want agents that represent us selectively on social media, that won’t disclose our weak negotiating position, that are good at sensing and currying power. Even if AIs didn’t acquire deception and power-seeking on their own, there’d be demand for them — which puts real weight on us, the consumers doing the selecting. Nathan adds the darker mechanism: throw models into cutthroat long-running environments where deception is rewarded — as Vending-Bench begins to show with price collusion — and you breed “seriously effective predatory AIs.” He recounts an Anthropic researcher explaining inoculation prompting and the emergent-misalignment generalization problem (a paper Nathan co-authored): reward a model for cheating and it learns to be a cheater broadly.Against this, Wright sets the noosphere — Teilhard de Chardin’s 1923 idea of a technologically-knit “global brain.” He sees genuine directionality (not necessarily purpose) in biological and cultural evolution, from self-replicating strands to cells to societies to the global community now taking shape. AI, he argues, is arriving exactly as that global brain forms — and some of its neurons may now be silicon. The question, echoing Nick Bostrom‘s “singleton,” is whether global coordination arrives the easy way (deliberate, decentralized, win-win) or the hard way (a coup, a seizure, a totalitarian nightmare). The title’s “God test” is not a claim that a god set this up; it’s that we face the kind of test gods are known for — we’ll have to become, in some sense, better people to pass it.Much of the back half is foreign policy, because Wright thinks the binding constraint on AI governance is psychological. He champions “cognitive empathy” — not feeling others’ pain, but understanding adversaries’ perspectives well enough to play non-zero-sum game