How to Construct Domain Specific LLM Evaluation Systems: Hamel Husain and Emil Sedgh

This talk focuses on architecting and testing reliable AI agents, moving beyond traditional chains and open-ended agents that suffer from poor reliability. The presenter introduces Lang Graph as a solution for building controllable agents that balance flexibility with robustness by expressing control flows as graphs with nodes and edges, incorporating state for memory.

This talk introduces BotDojo, an AI enablement company focused on helping businesses deploy AI applications to production. The demonstration highlights how to use synthetic data generation combined with evaluations to improve the performance of chatbots. It showcases a low-code editor for building AI flows, emphasizing the importance of detailed tracing for debugging and the integration of evaluations to identify issues like hallucinations and insufficient information retrieval.

This talk demonstrates how to build, evaluate, and deploy a Retrieval-Augmented Generation (RAG) based copilot for a retail environment using Azure AI services. The core concept involves creating a chatbot that can answer customer questions by accessing product information from a vector database and customer data from a relational database, all orchestrated through Azure AI Studio's Prompt Flow.

This talk addresses the critical and evolving landscape of AI evaluations, particularly as systems become more agentic and automated. It argues that current evaluation methods are often simplistic and prone to "benchmark hacking," failing to capture the true performance and values embedded in AI systems. The discussion emphasizes the need for more robust, multifaceted evaluation strategies that consider real-world scenarios, user experience, and the underlying values driving AI development to ensure these systems align with human goals.

This talk focuses on the practical challenges and solutions for building effective LLM evaluation systems in real-world applications. It distinguishes between model evals, which rank models against benchmarks, and task evals, which assess whether an LLM application is functioning correctly for its intended purpose. The core thesis is that robust task evals, especially those providing explanations for failures, are crucial for iterating and improving deployed LLM applications.