Europe 2026
Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind
Overview
This talk introduces the concept of "any-to-any" multimodal agents, enabled by Google DeepMind's Gemini API. The core idea is to build agents that can understand and generate across various modalities including text, code, images, audio, and video. The presentation outlines the architecture for such agents, focusing on multimodal understanding and native generation capabilities, and demonstrates how to build a notebook LM clone as an example.
Who should watch
- AI engineers and builders looking to create multimodal applications.
- Product Managers interested in the capabilities of advanced AI models.
- Developers seeking to integrate diverse data types (text, image, audio, video) into their AI projects.
- Anyone interested in building agentic systems that can reason and generate across different media.
Key takeaways
- Gemini models natively support multimodal understanding, accepting inputs like text, code, images, audio, and video.
- While Gemini's core model primarily outputs text, specialized native generation models exist for images, speech, and video.
- An agentic architecture can be built using Gemini as a reasoning model, which calls specialized tools for multimodal generation.
- The Google AI SDK simplifies multimodal agent development, allowing easy file uploads and integration with Gemini models.
- Gemini's audio transcription capabilities are robust, with significant token limits allowing for hours of audio input.
- Native generation models, like Nano Banana for images, leverage Gemini's world understanding for tasks such as correcting homework or drawing on maps.
- Function calling is crucial for enabling Gemini to interact with specialized generation models, allowing agents to decide when to create visuals or audio summaries.
- The live API offers real-time, audio-to-audio interaction, creating more natural conversational AI experiences.
Notable quotes
Patrick Löber states, *Gemini does not only understand text, right? It's natively multimodal, so you can also feed in code, image, audio, video, and then some more like URLs and also Google Search.*
Regarding audio input, *1 minute of audio translates to a 1920 tokens. And Gemini has a token limit of 1 million.*
On native generation, *these models understand the world.*
Unofficial community note. Prefer the recording for nuance.