WM@Booth

Almost every leap in modern AI has come from doing more things at once: parallel hardware, then parallel training, and now parallel generation. Diffusion models, long the standard for images and video, are bringing that same parallelism to language. Instead of writing one token at a time, they draft an entire answer and refine it over a handful of passes. In this talk, I will trace how diffusion language models went from a contrarian bet to a mainstream paradigm, then turn to what it takes to extend them across modalities. Training a single diffusion model to both understand and generate over images and text surfaces problems that neither modality raises alone: high-dimensional visual detail is costly to represent, language-vision correspondences are sparse and easily missed during training, and understanding and generation present different modeling tradeoffs. I will describe LaViDa, a line of work that addresses these challenges and offers a new basis for large-scale multimodal world modeling. The larger claim is that the frontier worth optimizing for world modeling is universal intelligence per watt: universality demands joint reasoning over modalities, efficiency demands parallelism throughout, and diffusion is one of the few paradigms that offers both.

The timescale of a robot's decision determines how much test-time computation it can afford, and what has to be compiled into the policy beforehand. I will illustrate this with three works from my lab. At the millisecond level there is no time to deliberate: in-hand tool reorientation and precise assembly require closed-loop reaction at 60 Hz. Therefore, a world model's role is to generate the rollouts a reactive policy is trained on rather than to be queried online. I illustrate this with SimToolReal and Play2Perfect. At seconds-scale, explicit prediction becomes tractable and useful: MessyNav decides which obstacles in a cluttered scene can be moved and where they will end up. This raises the question of what representation this actually needs, since it mixes semantic affordance with dynamics prediction and it is not obvious that a world model in the usual sense is the right abstraction. When dynamics are unknown, the useful test-time computation is adaptation rather than search: Causal-PIK refines a physics-informed model in-context from failed attempts, solving tasks in few trials. I argue that the timescale of a task determines when prediction should be paid for, at training time or at test time, while the form that prediction takes is determined by the task itself.

Language models have become powerful engines for reasoning, but many embodied problems depend on knowledge that is difficult to express purely in language. In this talk, I will present an approach to embodied reasoning in which generative world models provide a space for imagining and evaluating possible futures. I will show how video models can reason through generation, using inference-time search and temporal backtracking to revise unsuccessful or physically implausible predictions. These models can also be combined with vision-language models, linking semantic reasoning and high-level planning with spatially and physically grounded predictions. When conditioned on actions, world models can simulate candidate controls, enabling model-predictive control and grounding high-level instructions into continuous robot actions. Across visual reasoning, long-horizon planning, and robotic manipulation, these results suggest that world models can serve as a general interface for imagining, evaluating, and refining actions before they are executed.