World's Fair 2026
Frontier results, on device - RL Nabors, Arize
Overview
This talk explores the significant costs associated with using large, frontier AI models and proposes a strategy for migrating to smaller, more efficient, on-device models. The core thesis is that by right-sizing models for specific tasks, developers can drastically reduce expenses, improve user experience through lower latency, and enhance security and privacy, all while maintaining or even improving performance for many applications.
Who should watch
- AI Engineers
- Product Managers
- Developers building AI-powered features
- Anyone concerned with the cost, latency, or privacy implications of using cloud-based LLMs
- Teams looking to optimize AI inference for production environments
Key takeaways
- Using large, cloud-based foundation models incurs substantial costs related to security risks, user experience latency, unpredictable API expenses, and environmental impact.
- Smaller Language Models (SLMs) and task-specific models offer a viable alternative, consuming less energy and requiring significantly less computational power.
- The "prototype big, deploy small" methodology involves initially validating a concept with a powerful foundation model and then optimizing for production with smaller, specialized models.
- A structured evaluation process is crucial for selecting the right model, involving defining success criteria, collecting input-output pairs (a golden dataset), and testing models from smallest to largest.
- Tools like Arize's open-source Phoenix can be used to perform capability evaluations, comparing the performance of smaller models against larger ones on specific tasks.
- Prompt engineering, particularly techniques like few-shot prompting, can significantly improve the performance of smaller models, helping to close the gap with larger ones.
- On-device models offer benefits such as offline functionality, enhanced data privacy, and reduced inference costs by shifting computation to the user's device.
- Continuous evaluation, akin to CI/CD for code, is essential to prevent regressions and maintain the performance of deployed models and agentic workflows.
Notable quotes
*Prototype big. Deploy small.*
*The smallest model that gives acceptable responses for your use case.*
Unofficial community note. Prefer the recording for nuance.