188: World Models

Intro topic: RunningNews/Links:Flow matching versus diffusion modelshttps://youtu.be/firXjwZ_6KI?is=QMq8DcCsXTTktOuE OpenCV 5https://opencv.org/opencv-5/Claude fable beats pokemon with no harnesshttps://youtu.be/Ty_50J84fMY?si=EJ1KjZCZegipfCsV Can the stockmarket swallow Anthropic, SpaceX and OpenAI? https://archive.ph/nKEVwBook of the ShowPatrickStrength of the Few - James Islingtonhttps://amzn.to/4pmr10TJasonDescender - Jeff Lemirehttps://amzn.to/3QKhp3lPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrickNo Man’s SkyJasonPaperlib https://paperlib.app/en/ Topic: World ModelsMaking decisions with AIAction-Value (called a Q model): What is the long-term value of making a decision at a positionPolicy: What action should I take (must be a distribution)Value (called a V model): What is the value of a position (depends on policy)Advantage/Disadvantage: difference in value given two policiesWhen advantage is +, do that more.Model-FreeLook at the current situation and suggest an actionRun that action in the real world and measure the effectUse that measurement to suggest better actions next timeModel-BasedObserve rollouts (sequences of situations) and learn the dynamicsChoose an action, use your dynamics model to measure the consequencePotentially do MPC (try many actions and choose the best)World ModelsObserve many many rollouts and learn a full forward model (how to create the input in the future)Train a policy & value inside the world modelDeploy the policy and fine-tune based on the real world