TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

This talk argues that fundamental software engineering principles are more critical than ever in the age of AI. The speaker critiques the "specs-to-code" movement, which suggests generating code from specifications using AI, asserting that this approach often leads to deteriorating code quality. Instead, the talk emphasizes that well-structured, understandable, and maintainable codebases are essential for effectively leveraging AI's capabilities and that AI should be viewed as a tactical tool requiring strategic human oversight.

This talk addresses the escalating sophistication of AI attacks, moving beyond simple prompt injection to complex exploits targeting LLM interfaces, data, and even internal mechanisms. It proposes a practical, low-latency, self-hosted defensive layer built by fine-tuning a modern encoder model, specifically ModernBERT, to act as a guardrail against these threats. The approach emphasizes efficiency and cost-effectiveness, aiming for a defense under a dollar per instance.

This talk challenges the conventional understanding of "state-of-the-art" AI models, arguing that relying solely on public leaderboards or internal manual evaluations can lead to suboptimal choices. It emphasizes that true state-of-the-art is context-dependent and that efficiency, not just raw quality, is a critical factor in model selection for practical applications. The core thesis is that a more nuanced approach to benchmarking, considering specific use cases and efficiency metrics, reveals a landscape of multiple specialized, high-performing models rather than a single dominant one.

This talk explores the integration of coding agents, specifically the OpenClaw agent, into products. It emphasizes that coding agents are becoming a fundamental building block for software systems and highlights the Pi framework as an excellent tool for experimentation and development. The presentation demonstrates how agents can be leveraged to automate complex workflows, such as processing sales proposals, by interacting with various tools and systems.

This talk focuses on accelerating AI deployment on edge devices, highlighting Google DeepMind's Gemma models and the Lite RT framework. The core thesis is that by optimizing models for edge hardware and leveraging a unified cross-platform architecture, developers can achieve significant performance gains, enhanced privacy, and reduced latency for a wide range of AI applications.