New Google Model Ranked ‘No. 1 LLM’, But There’s a Problem

A new and mysterious Gemini model appears at the top of the leaderboard, but is that the full story? I dig behind the headline to show you some anti-climactic results, give some context with leaks in the last 48 hours of diminishing returns to scaling, and add the response of Altman, OpenAI and co. The future is about to look a lot stranger.80,000 hours Podcast + Channel: https://open.spotify.com/show/2WzJwXWBDnn4iZ7odKwDibhttps://www.youtube.com/@eightythousandhours/videos You can now gift memberships to AI Insiders (my Patreon w/ exclusive vids, network): https://www.patreon.com/AIExplained/giftChapters:00:00 - Introduction01:25 - LM Leaderboard02:35 - Benchmarks and Leaks05:31 - Low EQ07:37 - Other labs have issues too though10:31 - OpenAI claim and counter-claim14:13 - Other news‘There is no wall’: https://x.com/sama/status/1856941766915641580simple-bench.comhttps://x.com/vedantmisra/status/1857148554105544708Gemini Ranking: https://lmarena.ai/?leaderboardAPI not yet up: https://x.com/OfficialLoganK/status/1857106844805681153‘Just Die Chat’: https://x.com/koltregaskes/status/1856754648146653428https://gemini.google.com/share/6d141b742a13Google CEO tweet: https://x.com/sundarpichai/status/1857114106928718329Sutskever Quote: https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11/Another OpenAI Staffer Leaves: https://x.com/RichardMCNgo/status/1856843040427839804Bloomberg Report: https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai?s=09Noam Brown on what OpenAI Researchers Believe: https://x.com/polynoamial/status/1855037689533178289Clive Chan: https://x.com/itsclivetime/status/1855704120495329667Chollet Responds to Altman: https://x.com/fchollet/status/1857060079586975852https://x.com/sama/status/1856940152460869718Altman Emails: https://x.com/TechEmails/status/1857285960997712356Change of Heart: https://sd11.senate.ca.gov/news/senator-wiener-responds-openai-opposition-sb-1047Amodei on ‘Empirical Regularities’: https://lexfridman.com/dario-amodei-transcript/Verge Report: https://www.theverge.com/2024/10/25/24279600/google-next-gemini-ai-model-openai-decemberOpenAI Agents in January: https://www.bloomberg.com/news/articles/2024-11-13/openai-nears-launch-of-ai-agents-to-automate-tasks-for-users?srnd=phx-aiThe 8 Most Controversial Terms in AI: https://imp.i384100.net/m57g3MNon-hype Newsletter: https://signaltonoise.beehiiv.com/Podcast: https://aiexplainedopodcast.buzzsprout.com/I use Descript to edit my videos: https://get.descript.com/ldgxfuj2bhnb Learn more about your ad choices. Visit megaphone.fm/adchoices