✨ Why verification is AI's real bottleneck: My interview with economist Christian Catalini

On this episode of Faster Please!—The Podcast, I am joined by Christian Catalini. Christian is a research scientist and the founder of the MIT Cryptoeconomics Lab. Recently, Christian co-authored “Some Simple Economics of AGI” In our conversation, we focus on the AI bottleneck of verification, rising safety concerns, and how to address and solve these problems while maintaining growth and the US advantage over China.In this episode:* Red light or green light? (0:37)* AI risk and safety (8:07)* The bottleneck: verification (15:28)* Markets and AI safety (22:48)* AI labs and regulations (26:17)A lightly edited transcript of our conversation will appear in my Week in Review issue on Saturday. (Another option is using the Substack auto transcript function.)But here are some of the many highlights from the chat:✨ On slowing AI industry growth…Definitely not a red light for the entire space. I think that would be a very bad idea. That said, concerns have been raised by the labs themselves. So I think the next natural step is to send some real evaluators in and see what’s actually going on.We’ve seen a report from METR. I think they’re a talented team, but they have a very particular bias and a very particular type of expertise in this.I would love to see traditional security engineers and cyber experts looking at what happened inside OpenAI, what is happening inside Anthropic. Is there something we should be concerned about? Or are these companies just racing and potentially being a little bit reckless with their deployments?✨ On catastrophic AI risk…Could someone, on the bio side, use the information from an LLM to create something more dangerous than attempts that have been happening over the last few decades, where people were just ill-informed? They got 80 percent of the way. Could they get to 100 percent of the way and hurt a substantial number of people? Yes. Would it be a pandemic? I think that’s the big stretch.Once you do those massive jumps, I think that’s where we’re in the land of science fiction. And honestly, it’s harmful because we keep anthropomorphizing these agents when there are very benign explanations for what they’re doing. In fact, it’s the training that the labs are doing with reinforcement learning that’s leading exactly to the outcomes you’re seeing.✨ On the verification bottleneck…The act of verification is the actual bottleneck. Because with AI, you can just throw compute at anything that’s been measured and therefore can be automated. But if you can’t trust the output, if there are important unmeasured dimensions of that problem that the AI doesn’t have access to, well, then you’re in trouble.I think companies today, including the labs—and maybe this is why we’re seeing the security incidents—have this temptation to keep advancing and rapidly progressing, potentially not doing all the proper verification.✨ On regulating AI…I don’t think we need to come up with draconian, very tight regulatory bodies for mature markets, right? Airlines, financial services—how long did it take us to fine-tune that regulation so that we got it right? Decades. And in fact, it’s still a work in progress.I think here we need to be thoughtful. I think we can start with better measurement. Let’s surface the evidence.Let’s have neutral evaluators and inspectors. Again, we have METR, but we need probably 10 different organizations with very different philosophies, different skill sets, that can go in and provide us a neutral third-party assessment.By the way, neutrality also means funding. If the funding for these institutions comes from the labs, well, I don’t know what to make of it, right? Maybe academia has a role to play. I’m not sure, but we really urgently need better measurement.✨On the labs’ role in AI safety…The antitrust concerns don’t really apply in the format of safety. Of course, if you’re timing model releases and sort of handicapping the industry in different ways, that could be an antitrust concern. But I think many have said this very eloquently over the last few days. Yes, the labs are in full control. If they don’t believe their products are safe to ship, they don’t even need to coordinate, right? In a sense, each one of them can look at their own evidence and m