World's Fair 2025
What every AI engineer needs to know about GPUs — Charles Frye, Modal
Overview
This talk explains why AI engineers need to understand GPUs, shifting focus from API-based development to leveraging hardware capabilities. It draws an analogy to database usage, where developers don't build databases but must understand how to query them effectively. Similarly, AI engineers will increasingly need to understand GPU architecture, particularly tensor cores, to optimize performance for tasks like language model inference.
Who should watch
- AI Engineers
- Product Managers
- Builders working with AI models
- Those interested in optimizing AI workloads
- Developers looking to self-host and run their own models
Key takeaways
- GPUs prioritize high bandwidth and computational throughput over low latency, distinguishing them from traditional CPUs.
- The core of GPU performance lies in tensor cores, specialized for low-precision matrix multiplications, which are crucial for AI workloads.
- *Use the tensor cores Luke.* This means optimizing code to utilize these specialized units for matrix operations.
- Patterson's Law suggests bandwidth improvements consistently outpace latency improvements, making bandwidth-oriented hardware like GPUs a strategic choice.
- Arithmetic intensity, or performing many computations per memory load, is key to maximizing GPU efficiency, especially for tasks like language model inference.
- Self-hosting open-weight models is becoming increasingly viable due to improvements in open-source software and hardware capabilities.
- Techniques like running smaller models multiple times on the same prompt can improve efficiency and match quality by leveraging hardware strengths during decoding.
- Understanding GPU architecture, including concepts like parallel threads and rapid context switching, is essential for maximizing performance.
Notable quotes
The core thesis can be summarized as: *Use the tensor cores Luke.*
*Bandwidth has won out over and over again. So maybe bet on the bandwidth hardware.*
Unofficial community note. Prefer the recording for nuance.