Code 2025
Code World Model: Building World Models for Computation – Jacob Kahn, FAIR Meta
Overview
The Code World Model (CWM) project aims to build AI models capable of reasoning, planning, and decision-making, using code as a constrained environment for developing these capabilities. Instead of solely focusing on code syntax, CWM models program execution more explicitly by predicting program states and transitions. This approach seeks to bridge the gap between traditional world models and large language models, enabling more efficient agentic reasoning through simulated execution.
Who should watch
- AI Engineers
- Product Managers
- Builders working with code generation and analysis
- Researchers interested in agentic reasoning and world models
- Those facing challenges in debugging complex code or understanding program execution dynamics
Key takeaways
- CWM models program execution by predicting state transitions, moving beyond just syntax analysis.
- The project leverages a 32 billion parameter dense transformer trained on trillions of tokens, with extensive pre-training and fine-tuning.
- CWM is designed to be bash-oriented, learning to interact with the terminal to perform tasks, simulating an engineer's environment.
- The training process utilizes an asynchronous Reinforcement Learning (RL) setup to maximize throughput and efficiency, even updating models mid-trajectory.
- CWM demonstrates capabilities in tracing code execution line-by-line with high accuracy, including tracking local variable values.
- The model can function as a neural debugger, allowing users to express desired program structures and semantics loosely or precisely.
- CWM explores approximating solutions to complex computational problems like the halting problem by simulating program execution.
- The model and associated code are available on Hugging Face and GitHub to facilitate further research and development.
Notable quotes
*Our primary goal is to build models that reason, plan and make decisions.*
*We want to predict program execution because we believe it might lead to us better modeling things about code, writing code, analyzing code, and beyond.*
*With CWM, we're also taking a big step back with how we interact with the environment.*
Unofficial community note. Prefer the recording for nuance.