A year of Gemini progress + what comes next — Logan Kilpatrick, Google DeepMind

This talk focuses on mastering AI evaluation, moving from initial development to production. It emphasizes that even the best large language models (LLMs) require a robust testing framework due to issues like hallucinations and performance degradation with changes. Effective evaluation helps answer critical questions about model selection, cost-effectiveness, brand consistency, and ongoing improvement, ultimately reducing development time, costs, and enabling faster iteration.

This talk introduces the Google Gemini 2.5 model family, focusing on practical applications for AI engineers and builders. It highlights the multimodal capabilities of Gemini 2.5 Pro and Flash, demonstrating how to leverage these models for text generation, image and audio understanding, function calling, and integrating with external tools via MCP servers. The session emphasizes hands-on learning through a workshop format, encouraging attendees to experiment with the models and SDK.

This talk addresses the gap between AI-generated code prototypes and production-ready software. It introduces Sculptor, an experimental coding agent environment designed to build trust in AI-generated code by focusing on identifying and preventing defects. The core thesis is that AI should be leveraged not just for code generation but also for ensuring its quality and reliability.

This talk argues for building dynamic, deeply integrated AI products rather than AI "sideshows" that are bolted onto existing systems. The core thesis is that companies should move beyond using AI primarily to demonstrate technological capability and instead focus on solving customer problems by integrating AI strategically into their core product development. This approach requires aligning AI and product strategies, teams, and roadmaps, and embracing a crawl, walk, run methodology for iterative development.

This talk focuses on the practical aspects of building AI products that are reliable and effective, moving beyond theoretical discussions of evaluations. It emphasizes that while AI technology is advancing, the core challenge lies in effective communication and managing the inherent complexity and undefined behaviors of AI systems. The speakers advocate for an iterative approach to product development, where continuous refinement based on real-world user data and signals is crucial for success.