Europe 2026
From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google
Overview
This talk explores the development and application of small and tiny Large Language Models (LLMs) for on-device AI agents. It highlights the benefits of on-device processing, such as improved latency, privacy, and offline functionality. The presentation introduces the Google AI Edge stack, including MediaPipe and the lighter TLM runtime, and discusses two primary approaches: leveraging system-level GenAI like Gemini Nano and developing App GenAI with customizable tiny LLMs.
Who should watch
- AI engineers and developers interested in on-device AI.
- Product Managers and builders looking to integrate AI into mobile applications.
- Those exploring custom LLM solutions for specific tasks.
- Developers seeking to understand the capabilities and deployment of tiny LLMs.
Key takeaways
- On-device AI offers advantages in latency, privacy, offline use, reliability, and cost savings.
- The Google AI Edge stack includes MediaPipe, lighter TLM (a runtime for LLMs), and lighter T (a cross-framework runtime formerly TensorFlow Lite).
- System-level GenAI, like Gemini Nano via AI Core, provides pre-installed, highly optimized models for apps.
- App GenAI, using runtimes like lighter TLM, allows for greater customization with smaller, loadable models, though it requires more development effort.
- Agent skills can be developed and integrated into on-device LLMs, enabling specialized functionalities. The Google AI Edge Gallery app demonstrates this, allowing users to build and load custom skills.
- Tiny LLMs (under 1 billion parameters) have seen significant progress, making them viable for specific, narrow tasks when fine-tuned.
- Fine-tuning tiny LLMs, particularly for function calling, can dramatically improve accuracy, moving from 46% to over 90% success rates for specific tasks.
- The lighter TLM runtime supports exporting and running models in a single file format, compatible with various hardware accelerators like CPUs, GPUs, and NPUs.
Notable quotes
*The core idea is that we have system-level GenAI, which is something that will be pre-installed in the system.*
*If you want like more if you have a more specific task um that you want to do that's kind of highly customized or something really boutique, um you can use an App Gen AP Gen AI.*
*We've had a lot of success deploying those models internally and in an app a different app that you're going to see in a minute by doing kind of fine-tuning using synthetic data.*
Unofficial community note. Prefer the recording for nuance.