Europe 2026
Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind
Overview
This talk introduces the Gemini Interactions API, a new unified API designed to simplify building with large language models and agents. It emphasizes a more developer-friendly interface, akin to industry standards, and introduces server-side state management for improved agent development. The session also showcases the Gemini Live API for real-time conversational AI applications, including audio and video processing, and demonstrates how to build a coding agent with file I/O and bash command execution capabilities.
Who should watch
- AI engineers and developers looking to build conversational agents.
- Product Managers and builders interested in leveraging LLMs for new applications.
- Developers seeking to integrate real-time audio and video processing into AI applications.
- Those interested in agent development, tool use, and function calling with LLMs.
- Engineers exploring new APIs for model interaction and agentic workflows.
Key takeaways
- The Gemini Interactions API offers a unified interface for models and agents, simplifying API calls and aligning with common industry patterns.
- Server-side state management in the Interactions API reduces the need for client-side history management and improves cache hit rates.
- The Gemini Live API enables real-time, stateful interactions with models, supporting streaming audio, video, and text, with applications in areas like customer support and interactive experiences.
- Building coding agents can be facilitated using agent skills, allowing models to reliably perform tasks like reading/writing files and executing bash commands.
- The Gemini Live API's native audio processing allows for direct sound token to sound token interaction, supporting multilingual capabilities and reducing latency compared to cascaded pipelines.
- Tool calling and function integration are core features, enabling agents to interact with external services and custom functions, with improved capabilities for combining tools like Google Search with custom logic.
- The Gemini 3 Flash model, despite its speed and cost-effectiveness, can be effectively used for coding tasks when provided with good instructions, skills, and context.
- Real-time applications using the Gemini Live API can be built using either server-to-server architectures or direct client-to-API connections via ephemeral tokens.
Notable quotes
*The Interactions API is a new API we launched in December in beta, which hopefully will succeed Generate Content soon.*
*We want to build this unified interface where you can continue your conversations, no matter what model you used.*
*The Gemini Live API is a stateful kind of web socket API. You are able to send real-time text, audio, video feeds to the model.*
Unofficial community note. Prefer the recording for nuance.