Session brief

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Cormac Brick

Overview

This talk explores the challenges and opportunities of running AI models on edge devices, where memory (RAM) is the primary constraint, not compute power. It highlights the development of smaller, more efficient models, including quantized versions of Gemma and even sub-billion parameter models, to enable AI capabilities on resource-limited hardware like Raspberry Pis and mobile NPUs. The focus is on practical applications and the trade-offs involved in deploying AI at the edge.

Who should watch

• AI engineers working with resource-constrained environments.
• Developers interested in edge AI, robotics, and offline applications.
• Product managers evaluating the feasibility of AI features on edge devices.
• Builders looking for efficient model deployment strategies.
• Anyone concerned with the increasing cost and decreasing availability of RAM in consumer electronics.

Key takeaways

• The primary limitation for edge AI is not processing power but available RAM, which is becoming more expensive and less abundant in devices.
• Quantization techniques, such as reducing Gemma to 2.9 bits per weight, enable larger models to run on devices like Raspberry Pi, achieving speeds of approximately 8 tokens per second.
• Tiny models, ranging from 500 million down to 50 million parameters, are crucial for older laptops and low-cost devices where even small models are too large.
• Fine-tuning these tiny models, rather than just prompting, is often necessary to achieve desired performance, enabling capabilities like reliable voice-to-function calling.
• A practical application demonstrated is an offline voice dictation app built using two sub-billion parameter Gemma models, which also handles filler words.
• The Google AI Edge team provides resources like the AI Edge Gallery to facilitate experimentation with these small models on various hardware.
• Running AI at the edge allows for offline functionality, reduced latency, and enhanced privacy compared to cloud-based solutions.
• The trend of decreasing RAM in new devices necessitates a continued focus on model compression and efficiency for edge AI.

Notable quotes

Watch on YouTube →

Unofficial community note. Prefer the recording for nuance.