Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs

Code-as-World is a system that converts real videos into executable physics simulations by representing scenes as code rather than images or descriptions. The approach argues that video pixels show evidence of physical events but do not capture the underlying physics like mass, gravity, and collisions, which are needed for accurate simulation and training. The system uses an iterative loop that proposes a physics program, executes it, renders the result, and compares it against the original video across multiple rounds to refine accuracy. This verified simulation data provides exact physical labels that real video alone cannot supply, allowing machine learning models trained on it to score higher on physics reasoning benchmarks than comparable systems.