Dissertation Talk: Inferring Structured Physics World Models from Videos

Event details Date: Wednesday, August 5, 2026 Time: 1:00 PM to 1:00 PM Location: Gateway Room 4350 (4th-Floor Social Kitchen) Type: Lecture / Workshop Audience: Faculty,Students About this event Despite remarkable progress in deep reinforcement learning (RL), sample efficiency remains a bottleneck: modern agents require orders of magnitude more experience than humans to learn even simple tasks. A five-year-old child can perform novel manipulation tasks zero-shot, suggesting that human physical reasoning might be grounded in something richer than the large-scale imitation learning that powers our best current systems. Cognitive science offers a compelling hypothesis, that humans reason about the physical world by performing probabilistic mental simulations, actively running experiments and updating structured, causal models of their environment, much like a scientist or a hacker probing a system. This thesis investigates how to build structured, interpretable world models directly from observations, framed as a problem of probabilistic programming and Bayesian inference. Unlike differentiable neural architectures, structured models are generally non-differentiable, making inference over both model structure and parameters challenging. We begin by characterizing the failure modes of existing probabilistic inference algorithms in this setting. We then build a novel framework, utlizing Metropolis-adjusted Neural Proposals, which augments classical MCMC with a learned neural proposal distribution, and study the challenges that arise when applying it to structured physics models: the sparsity of priors, the joint inference of model structure and parameters, and scaling to richer domains. We develop techniques to mitigate each of these, yielding a general framework for inference over structured models. We demonstrate the framework across a range of tasks: inverse graphics, program synthesis for the ARC-AGI challenge, and physical reasoning in 2D physics environments, where our approach achieves sample efficiency an order of... Official event details: https://events.berkeley.edu/eecs/event/325007-dissertation-talk-inferring-structured-physics