Code 2025
Compilers in the Age of LLMs — Yusuf Olokoba, Muna
Overview
This talk addresses the fundamental challenges AI engineers face in deploying models, moving beyond the hype of voice agents and MCP tools. The core thesis is that building a Python compiler to generate self-contained binaries offers a simpler, more standardized approach to running AI models anywhere, from local devices to the cloud, without extensive infrastructure changes. This method aims to replicate the ease of using models like OpenAI's API but with the flexibility of any open-source model.
Who should watch
- AI Engineers struggling with model deployment complexity.
- Developers needing to integrate diverse AI models into applications.
- Teams looking to enable hybrid inference with local and cloud models.
- Builders seeking to simplify the process of running models across different hardware.
- Product Managers evaluating deployment strategies for AI features.
Key takeaways
- Current AI deployment often involves complex infrastructure setup for each new model, such as writing Dockerfiles and managing cloud resources.
- The goal is to create an open-style client that works with any model, regardless of its location or framework, requiring minimal code changes.
- A Python compiler was developed to convert plain Python inference code into a self-contained binary executable on any platform.
- The compiler pipeline uses LLMs for code generation and verification, alongside LLM-powered testing, to ensure robustness.
- The process involves tracing Python functions to create an intermediate representation (IR), then translating this IR into lower-level C++ or Rust code.
- Type propagation is a key technique used to infer and constrain variable types from dynamic Python to static C++/Rust.
- LLMs are leveraged to generate the necessary C++ or Rust code for elementary operations, making the translation process tractable.
- The compiled binaries can be invoked from various environments, including JavaScript via Node.js using FFI, and can be exposed through an OpenAI-style client interface.
Notable quotes
*What developers actually want is something way simpler. Just give me an open style client that just works.*
*We wanted to recreate something that tracked this experience as closely as possible.*
*The only reason why that's tractable now is well we don't have to sit down and write the equivalent native code that does the same thing in Python anymore.*
Unofficial community note. Prefer the recording for nuance.