World's Fair 2025
Building Multimodal AI Agents From Scratch — Apoorva Joshi, MongoDB
Overview
This talk introduces the concept of building multimodal AI agents from scratch, focusing on integrating text and image processing capabilities. It explains the evolution from simple prompting and RAG to AI agents, highlighting agents' suitability for complex, multi-step tasks requiring reasoning and action. The session details the core components of an agent—perception, planning/reasoning, tools, and memory—and demonstrates how to construct a multimodal agent capable of answering questions about documents containing both text and images, and analyzing charts or diagrams.
Who should watch
- AI engineers and developers interested in building advanced AI applications.
- Product Managers and builders looking to incorporate multimodal AI into their products.
- Individuals seeking to understand the practical implementation of AI agents.
- Those facing challenges with processing and retrieving information from documents containing mixed media (text, images, tables).
Key takeaways
- AI agents use LLMs to reason, plan, and execute tasks with the help of tools, making them suitable for complex, multi-step problems.
- Multimodality in AI refers to the ability of models to process and generate various data types like text, images, and audio.
- Vision-language models (VLMs) simplify multimodal data processing by using a unified encoder for text and images, preserving contextual relationships and improving retrieval quality.
- Building a multimodal agent involves preparing a corpus by converting documents into screenshots, embedding them using a multimodal embedding model, and storing them in a vector database.
- Agents utilize a combination of multimodal embedding models for retrieval and multimodal LLMs for reasoning and generating responses that incorporate both text and visual information.
- Short-term memory can be implemented by storing and retrieving chat history associated with a session ID, allowing for coherent multi-turn conversations.
- The React (Reasoning and Acting) pattern is a common design for agents, where the LLM generates reasoning traces and actions, observes tool outcomes, and iterates until a final answer is reached.
- When dealing with mixed-modality documents, a common approach is to take screenshots of each page to preserve the continuity between text and image elements for retrieval.
Notable quotes
Apoorva Joshi defines an AI agent as a system that uses an LLM to reason through a problem, create a plan, and execute it with the help of tools.
The talk emphasizes that agents are best used for complex tasks that don't have a structured workflow or where the series of steps is hard to predict.
Unofficial community note. Prefer the recording for nuance.