Who Tests the Test?

Peer review has been the cornerstone of scientific quality control for decades—but it's actually younger than rock and roll, and it's failing catastrophically. Joey examines the numbers: 36% of psychology studies fail to replicate, failed experiments get cited 150x more than successful rebuttals, and the filter keeps getting worse. This episode explores why Magic: The Gathering's Sealed Deck format is a better model, and what open peer review, post-publication review, and registered reports could actually fix. 00:00:00 - Peer review history and the replication crisis 00:01:41 - 36% of psychology studies fail to replicate 00:04:58 - Magic: The Gathering's Sealed Deck as a metaphor for systems 00:08:33 - Post-publication review: the model that could work 00:10:17 - Why open peer review fails against institutional gatekeepers 00:13:41 - What the speedrunning community got right about verification 00:15:24 - Institutional gatekeepers and why they won't budge --- Sources & further reading: • Melinda Baldwin — "In Referees We Trust?" — Physics Today (2018): https://physicstoday.aip.org/features/in-referees-we-trust • Anne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens — "An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports" — Advances in Methods and Practices in Psychological Science (2021): https://research.tue.nl/en/publications/an-excess-of-positive-results-comparing-the-standard-psychology-l-2/ • Open Science Collaboration — "Estimating the Reproducibility of Psychological Science" — Science (2015) • Daniele Fanelli — "Positive results increase down the hierarchy of the sciences" — PLOS ONE (2010) • eLife Sciences — "eLife's New Model: One Year On" (2024): https://elifesciences.org/inside-elife/66d43597/elife-s-new-model-one-year-on • Kochetkov — "Peer Review at the Crossroads" — Learned Publishing (2026): https://onlinelibrary.wiley.com/doi/full/10.1002/leap.2046 • PLOS Blog — "Pioneering Approaches in Open Peer Review" (2025): https://theplosblog.plos.org/2025/09/pioneering-approaches-in-open-peer-review/ • Cheng, Yang, and Allen — paper mill retraction analysis via Retraction Watch Database (2024): https://retractionwatch.com/2025/08/04/fighting-coordinated-publication-fraud-is-like-emptying-an-overflowing-bathtub-with-a-spoon-study-coauthor-says/ • Pangram Labs / Pebblous AI — "When AI Reviews AI: 21% of ICLR 2026's 76,139 Peer Reviews Were AI-Generated": https://blog.pebblous.ai/report/iclr-2026-ai-peer-review-crisis/en/ • Xu and Waltman — "Streetlight Effect in Post-Publication Peer Review: Are Open Access Publications More Scrutinized?" (2023 preprint): https://arxiv.org/pdf/2311.04909 • Nature survey of 1,576 researchers on reproducibility (2016) • SciCampus — "Retraction Crisis 2026: AI Paper Mills and Fake Citations": https://scicampus.com/news/retraction-crisis-2026-ai-paper-mills-hallucinated-citations • Chambers and Tzavella — census of Registered Reports journals (2022): https://online.ucpress.edu/collabra/article/7/1/24404/117300/Opening-the-Door-to-Registered-Reports-Census-of • Sciopen — "Academic Community's Perceptions of Registered Reports" (2026): https://www.sciopen.com/article/10.16510/j.cnki.kjycb.20260526.001 • Springer/Scientometrics — "Scientific publishing without gatekeeping: an empirical investigation of eLife's new peer review process" (2025): https://link.springer.com/article/10.1007/s11192-025-05422-y • PubPeer Wikipedia entry: https://en.wikipedia.org/wiki/PubPeer • ResearchGate — "Public post-publication peer review needs due process" (2024): https://www.researchgate.net/publication/410988280_Public_post-publication_peer_review_needs_due_process_accountability_expertise_and_fairness_in_the_PubPeer_era • …and 2 more in the episode research notes This podcast episode was fully generated by AI — research, script, voices, and production. Built with Claude, Piper TTS, and automated pipeline tooling.