Apoorva Joshi

This talk outlines a repeatable framework for designing and building AI systems from initial idea to production. It emphasizes that in the current AI landscape, defining product requirements, system design, and evaluation criteria are the critical, challenging aspects, rather than just coding. The framework consists of four phases: product requirements, system design, evaluation and monitoring, and optimization.

This talk introduces the concept of building multimodal AI agents from scratch, focusing on integrating text and image processing capabilities. It explains the evolution from simple prompting and RAG to AI agents, highlighting agents' suitability for complex, multi-step tasks requiring reasoning and action. The session details the core components of an agent—perception, planning/reasoning, tools, and memory—and demonstrates how to construct a multimodal agent capable of answering questions about documents containing both text and images, and analyzing charts or diagrams.