World's Fair 2025
Hacking the Inference Pareto Frontier - Kyle Kranen, NVIDIA
Overview
This talk explores techniques for optimizing AI model inference to break the Pareto frontier, focusing on balancing quality, latency, and cost. The core thesis is that a well-designed system, tailored to specific application constraints, is crucial for successful deployment and application performance. By understanding and manipulating factors like scale, structure, and dynamism, engineers can achieve better service level agreements or reduce costs for existing ones.
Who should watch
- AI Engineers
- Product Managers
- Builders working with large language models
- Those looking to optimize inference costs and performance
- Engineers facing challenges with model latency and quality trade-offs
Key takeaways
- The inference Pareto frontier represents the trade-off between performance metrics like tokens per second per GPU (cost) and user tokens per second (responsiveness).
- Application requirements dictate the optimal operating point on the Pareto frontier; for example, personal cancer cures have different latency and cost tolerances than code completion.
- Common techniques like quantization and retrieval-augmented generation can be compounded to manipulate performance characteristics, though they may have opposing effects on latency and quality.
- Disaggregation, separating prefill and decode phases of inference, can significantly improve performance by allowing granular load matching and reducing scheduling conflicts, especially for interactive applications.
- Smart routing, which considers KV cache locality and worker load, can increase speed and reduce cost without impacting quality.
- Structuring workloads, such as through inference time scaling (re-querying models), can improve quality or, conversely, maintain quality while reducing latency and cost by using smaller models.
- Dynamism, through worker specialization and dynamic load balancing, is essential for adapting to changing user distributions and maximizing the effectiveness of techniques like disaggregation.
- NVIDIA Dynamo is an open-source project aimed at enabling data center scale inference and manipulating the Pareto frontier.
Notable quotes
*A good model and a good system that takes into account the actual constraints for what you need from your deployment is actually key to the success of both your deployment and the application that is backed by it.*
*The three things that we or I like to think about when I'm thinking about whether or not something can actually be deployed and used is really simple. It's quality. Latency. And cost.*
Unofficial community note. Prefer the recording for nuance.