World's Fair 2025
Continuous Profiling for GPUs — Matthias Loibl, Polar Signals
Matthias Loibl , Polar Signals
Overview
This talk introduces continuous profiling for GPUs, a method to monitor and analyze GPU performance. It highlights the importance of profiling for improving application performance and reducing operational costs by enabling more efficient resource utilization. The approach leverages Linux eBPF for low-overhead, always-on profiling in production environments without requiring application instrumentation.
Who should watch
- AI Engineers
- ML Engineers
- Performance Engineers
- Builders optimizing GPU-intensive applications
- Anyone seeking to understand and improve GPU utilization
- Teams looking to reduce cloud infrastructure costs
Key takeaways
- Profiling, a long-standing technique in computer science, is crucial for performance engineering, aiming to understand time spent on CPUs and GPUs, memory usage, and function call frequencies.
- Continuous profiling, particularly sampled profiling, offers a low-overhead method to gather performance data in production, balancing detailed insights with manageable data volume.
- The presented solution utilizes Linux eBPF, allowing for system-wide profiling without modifying applications, thus simplifying deployment and integration.
- GPU profiling capabilities include monitoring overall node utilization, individual process activity, memory utilization, clock speeds, power consumption, temperature, and PCIe throughput.
- Correlating GPU metrics with CPU profiles, generated via eBPF, helps identify bottlenecks where the CPU might not be effectively keeping the GPU busy.
- A new feature, GPU time profiling, directly measures the time individual CUDA functions spend executing on the GPU, providing deeper insights into kernel performance.
- The tools support various languages and applications, including Python, Rust, C++, Ruby, and JVM-based applications, as well as web servers and databases.
- Getting started involves deploying a binary or daemon set for Kubernetes, requiring a token for data ingestion.
Notable quotes
*Profiling is pretty much as old as programming.*
*We want to see the big picture.*
*It is pretty important to be able to do this in production and actually see what's happening out there in the real world and do it with low overhead.*
Unofficial community note. Prefer the recording for nuance.