World's Fair 2025
Waymo's EMMA: Teaching Cars to Think - Jyh Jing Hwang, Waymo
Overview
This talk explores the evolution of autonomous driving systems, highlighting the transition from early, less capable models to sophisticated L4 systems like Waymo's. It introduces EMMA, an experimental system leveraging multimodal large language models (LLMs) like Gemini to enhance driving capabilities, particularly in handling rare and complex scenarios. The core thesis is that LLMs can significantly improve the generalizability and safety of autonomous driving by understanding and reacting to diverse real-world situations.
Who should watch
- AI Engineers
- Product Managers
- Autonomous Vehicle Researchers
- Machine Learning Practitioners
- Builders working on complex AI systems
Key takeaways
- Early autonomous driving research in the 1980s used simple neural networks, while modern systems like Waymo's L4 vehicles employ complex perception, prediction, and planning modules.
- Scaling autonomous driving to new cities and diverse conditions presents significant challenges, especially with "long-tail" events that are rare but critical.
- Foundation models, particularly multimodal LLMs like Gemini, show promise in generalizing to these rare events, understanding complex scenarios, and providing appropriate driving responses.
- Waymo's EMMA system uses Gemini to translate routing information and camera input into driving decisions, outputting future waypoints without requiring high-definition maps.
- EMMA's initial formulation achieved state-of-the-art performance on the NuScenes benchmark and demonstrates self-supervised learning, camera-only input, and map-free operation.
- A "channel-so" reasoning approach within EMMA allows the model to explain its driving decisions by identifying critical objects and predicting their behavior, leading to improved performance on larger datasets.
- The system is being further generalized to handle multiple tasks beyond driving, including 3D detection and free-form visual question answering, demonstrating its versatility.
- Robust evaluation, including simulation using generative models for sensor data, is crucial for validating and ensuring the safety of these advanced autonomous driving systems.
Notable quotes
*The secret is very basically on the top where you can see that this is a sort of visualization or departing screen where it visualize everything that the system understands.*
*One of the the the the solutions arises where foundation model is is very generalizable.*
*The questions becomes like how how do we like how do we leverage this technology for Thomas driving.*
Unofficial community note. Prefer the recording for nuance.