How to Build Frontier-Lab Quality Evals with Daniel McKinnon, ex-PM at Meta, Google

Today’s EpisodeA developer posted this workflow in March, and it is the clearest picture of where PM is heading that I’ve seen all year.Rasty Turek spent the past year building with coding agents, and he mapped how his process changed over that time. He reckons he now spends around 90% of his time on evals. His eval started as QA, and then it became the spec.Great, now everyone agrees evals are important and will become indispensable for PMs going forward. But there is very little on how to write one.That changes today.I’ve now done 6 episodes on evals, and all of them start with an agent that is running and failing. So what do you do on day 0?Daniel McKinnon was a PM on the Llama models at Meta, a boomerang who spent around 7 years there in total. He sat on Facebook’s central AI team for the entirety of its existence. He wrote enterprise evals for Gemini, Llama, and Ray-Ban Meta.His first job at Meta was on the speech recognition team. He had to figure out how to check whether the models were any good. They weren’t called evals back then. But he’s been writing them for his entire career anyway.In this episode you’ll learn:* How to build an eval set from nothing* The floor-and-ceiling method for calibrating* How to score it and make the shipping callCheck it out:Please fill out this short survey on PM salaries.🆓 I’m doing a free webinar Thursday on getting AI PM interviews. Join me:The next cohort of my LandPMJob program starts in August. If you want my 1:1 coaching, sign up.----Check out the conversation on Apple, Spotify, and YouTube.Brought to you by:* SerpApi - Get started with SerpApi using 250 free credits.* Product Faculty - Get $550 off their #1 AI PM Certification with code AAKASH550C7* Ariso - Ship AI agents and features faster, with fewer regressions* Land PM Job - 12-week experience to master getting a PM job* Pendo - The #1 software experience management platform----Key Takeaways:1. An eval is a trivia question for the model - At its core, an eval is a prompt with a correct or plausibly correct answer plus a way to score whether the output is good. It is the clearest way to communicate what your product should do in the AI era.2. Offline evals catch problems before you ship - Test the model offline against a fixed prompt set before pushing to production. If it fails, you change the model, the prompt, or the approach before real users ever see it.3. The best eval sits between too easy and too hard - An eval that scores 100% gives your engineering team nothing to optimize. An eval that scores 0% is equally useless. Aim for a 25% to 50% success rate so there is room to run.4. Old benchmarks are already saturated - MMLU, HellaSwag, ARC and the rest were built for a simpler question-and-answer world. Frontier models now score effectively 100% on them, which is why you have to keep building new evals and throwing away old ones.5. Writing an eval is mechanical once you understand the problem - Come up with roughly 100 prompts that match the real distribution of tasks. The hard part is not the writing. It is deeply understanding the domain first.6. Subject matter expertise drives everything - The cystic fibrosis and congenital heart disease evals worked because Daniel understood the genetics, not because of any template or tool. There is no eval template the way there is a PRD template.7. Modern evals are agentic, not just Q&A - The genetics eval hands the agent a file with billions of variants and as