Laurie Voss

This talk focuses on the practical aspects of evaluating and improving AI agents, moving beyond simple "vibe checks" to implement robust testing frameworks. It emphasizes that effective evaluation is crucial for shipping reliable AI applications, especially agents, which are prone to cascading failures. The session introduces methods for capturing agent behavior through tracing, categorizing failures, and implementing various types of evaluations (code-based, LLM-as-judge, human) to ensure agents perform as expected and to drive iterative improvements.

This talk introduces LlamaIndex as a framework for building generative AI applications, with a particular focus on agents. It emphasizes the necessity of Retrieval Augmented Generation (RAG) for agents to effectively process and utilize large, unstructured datasets. The presentation outlines several high-level design patterns that enhance agent performance in production environments.