Four Months Inside a Production AI Agent: What Real Users Changed

When ML/AI Engineer William Horton last joined me, Maven Assistant had reached its first external users the day before. The healthcare AI agent was available to 20 percent of Maven Clinic’s users, and the team had deliberately withheld answers about benefits. A wrong response could shape a decision involving $15,000 of fertility coverage, and the evals had not earned the right to ship it.Four months later, Maven Assistant is available to 100 percent of users, benefits answering is live, and weekly conversation volume has grown by roughly ten times. Real usage also overturned part of the roadmap. The team had invested heavily in provider search and appointment tools, but 50 to 60 percent of early conversations were basic health questions such as whether someone could eat tuna while pregnant.Production changed the engineering system too. An emergency guardrail told someone already in the ER to go to the ER. Zendesk content told people already using the Maven app to open the app. A newer model failed an upcoming-appointments eval because it correctly noticed that the mocked appointments were in the past.William explains how Maven turns those failures into deterministic tests, LLM judges, synthetic negatives, and manual review. He also walks through the move from Gemini Flash models toward newer OpenAI models, what GPT-5.6 and Fable mean for a production agent, why model upgrades can make old prompt instructions obsolete, and why open-weight models still have to justify their GPU, infrastructure, and engineering costs.“If anybody tells you that they’ve got their evaluations so good that they can just swap a model and know, with no manual review, that it’s going to be better, that person is probably lying, or they work at one of three places in the world.”— William Horton, Staff Machine Learning Engineer, Maven ClinicYou can also find the full episode on Spotify, Apple Podcasts, and YouTube.👉 Want to build agents from the ground up? Registration is open for Build AI Agents from First Principles, a live workshop on the loops, tools, context, harnesses, and engineering decisions behind useful AI agents. You’ll learn how to design agent systems from first principles, with enough structure to decide which harness patterns your product actually needs. Sign up today with code production10 for 10% off. 👈In This Episode* What changed between 20 percent and 100 percent rollout. Benefits answering cleared its release bar, Maven Assistant reached the remaining users, and weekly conversation volume grew by roughly ten times.* Why real usage beat the original roadmap. The complex provider and appointment agents received less traffic than expected, while 50 to 60 percent of early conversations were basic health questions.* How production failures enter the evaluation system. William estimates that the deterministic tool-use layer now involves more than 1,000 test scenarios, while clinical quality, empathy, and completeness still require judges, human calibration, and manual work.* When the model is right and the eval is broken. GPT-5.6 Terra rejected an “upcoming” appointment that had already happened, exposing a bad test fixture that the previous model had accepted.* What happens when GPT-5.6 or Fable arrives. A model swap can remove old prompt instructions, add new behavioral failures, change latency and cost, or reveal that yesterday’s harness is constraining a more capable model.* Why Maven moved beyond Gemini Flash 2.5. William discusses adopting newer OpenAI models, keeping real-time chat on smaller models and low reasoning settings, and changing the model without simultaneously rewriting the prompt.* The economics of open-weight models. A smaller self-hosted model still needs an always-on GPU, infrastructure, and engineering attention that could otherwise go into the product.* What William would rebuild today. Provider search and appointment booking probably belong in one agent, and model experiments should begin before the original choice hardens into the architecture.Start With the First EpisodeWilliam first joined Vanishing Gradients the day after Maven Assistant reached external users. In Building an Enterprise AI Agent for Healthcare, he explains the original architecture, how failures become regression cas