Evals & quality

The State of AI Engineering 2026 survey highlights evaluation (evals) as the most significant challenge across AI stacks. This talk delves into the survey results, emphasizing the critical need for robust evaluation methodologies in AI development. Addressing evals is presented as paramount for building reliable and high-quality AI systems.

This talk focuses on the critical importance of building effective evaluation systems for AI models, moving beyond superficial metrics to create benchmarks that genuinely reflect real-world performance and user needs. The presenters emphasize that robust evals are essential for shipping reliable AI products and driving meaningful improvements in model quality.

This talk addresses the common problem of AI evaluations that fail to predict real-world performance. The core thesis is that offline evaluations often use simplistic "customers" and test sets that don't reflect the complexity and adversarial nature of actual user interactions, leading to shipped models that fail in production. A solution involves building more realistic, adversarial user simulators trained on real data.

This talk introduces Vending-Bench, a novel evaluation framework designed to assess long-horizon agentic behavior in simulated business environments. It addresses the challenge of models acting differently when they suspect they are being tested, proposing methods to maintain reproducibility and observe emergent behaviors like collusion and power-seeking over extended periods. The work also explores the transition of AI-run businesses from simulation to the real world.

This talk emphasizes the critical need for rigorous evaluation of AI skills before deployment, arguing that shipping skills without proper testing can lead to unpredictable behavior and degraded performance. The speaker highlights the distinction between agents developers use for personal productivity and agents built for consumers, noting that end-users lack the context to troubleshoot skill invocation issues. The core thesis is that comprehensive evaluations are essential for ensuring skill reliability, managing costs, and determining when skills can be retired as models improve.

This talk proposes a shift from traditional, simplistic model evaluation methods to more sophisticated techniques borrowed from psychology and psychometrics. The core argument is that simply counting correct answers, a method akin to classical test theory, is insufficient. Instead, the presentation advocates for Item Response Theory (IRT) to provide a more nuanced understanding of model capabilities by calibrating individual questions and estimating model intelligence more accurately.

This talk addresses the significant gap between the rapidly advancing reasoning capabilities of large language models (LLMs) and the slower evolution of retrieval systems. The core thesis is that LLMs are bottlenecked not by their reasoning power, but by their ability to access the correct knowledge. By improving retrieval tools and teaching agents to use them effectively, most of this knowledge gap can be closed, unlocking LLMs for complex tasks beyond coding.

This talk introduces SWE Marathon, a benchmark designed to evaluate the capabilities of coding agents on large-scale, project-level tasks. It addresses the growing need to assess if agents can maintain coherence and successfully complete complex engineering projects over extended operational periods, such as a billion token budget, moving beyond simple bug fixes to end-to-end project ownership.

This talk addresses a critical failure point in AI agents: the inability of evaluation signals to inform future actions, leading to repeated errors. The core thesis is that current agent memory systems are static and do not learn from past successes or failures. The presentation introduces a new runtime experience, Agent RX, designed to bridge this gap by allowing agents to improve dynamically based on outcomes without retraining or manual prompt engineering.

This talk argues that traditional benchmarks are insufficient for evaluating agentic AI systems in production. Agentic systems, which plan, call tools, and execute workflows, require a shift in evaluation focus from model capability to system behavior. The core thesis is that production telemetry and reliability metrics are paramount for dependable outcomes, moving evaluation from a pre-deployment phase to a continuous operational capability integrated into the system's control plane.

This talk argues that current evaluation methods for role-playing language agents (RPLAs) are fundamentally flawed. These methods, which prioritize fluency and personality consistency, fail to detect a dominant failure mode: anachronistic compositing. This occurs when models generate personas that sound like historical figures but reason using knowledge and moral frameworks that postdate them, often due to the overwhelming influence of culturally dominant, later representations in training data. The proposed solution is a shift towards epistemic simulation, emphasizing corpus-boundedness, temporal anchoring, and expert-loop evaluation.

This talk introduces Eval++, a new compute primitive designed to enhance the development and deployment of AI agents. The core idea is to leverage Cloudflare's stateful serverless technology, specifically Durable Objects, as an ideal execution environment for agents. This approach enables long-running processes, persistence, and efficient scaling, fundamentally changing how developers can build and interact with AI-powered applications.

This talk introduces a platform for LLM observability, evaluation, and experimentation, emphasizing that building AI systems is fundamentally an engineering discipline. It highlights the importance of understanding what an AI system is doing (observability), how to measure its performance (evaluation), and how to systematically improve it (experimentation). The core thesis is that these processes, while complex in the non-deterministic world of AI, can be managed and eventually automated through robust engineering practices and tooling.

This talk critiques the current state of AI model evaluations, arguing that while flawed, they remain essential tools for development. The speaker proposes a three-stage approach to using evals effectively: leveraging external evals, using them to improve internal agents, and building custom evals for specific use cases. The core message is to use evals pragmatically, understanding their limitations while still employing them for iterative improvement.

This talk addresses the growing gap between the capabilities of AI agents and the methods used to evaluate them. It argues that while progress is evident in areas like coding agents, current evaluation benchmarks are falling behind, creating hesitation in deploying these agents in high-stakes environments. The presentation outlines key principles for building effective benchmarks, distinguishing between the scientific rigor required for measurement and the artistic vision needed to shape the future of AI development.

This talk details the practical lessons learned from evaluating coding agents on real-world software engineering tasks using the rebench leaderboard. It emphasizes the critical need for robust evaluation beyond gut feelings or limited testing, especially as models are deployed to production. The presentation highlights the challenges and methodologies involved in creating and maintaining a "fresh" and "real-world" benchmark, focusing on the complexities of software engineering tasks that require understanding repository structures, writing and running tests, and handling multi-turn interactions and tool use.

This talk explores the effectiveness of semantic code retrieval for AI coding assistants, specifically benchmarking it against traditional grep-based methods on Claude Code. The core thesis is that while grep is simpler and often sufficient, semantic search, powered by vector embeddings, can significantly improve precision and the agent's ability to locate relevant code, especially for complex tasks. This approach offers a form of cached compute, reducing redundant processing and potentially leading to better accuracy and user satisfaction.

This talk addresses the quality of code generated by Large Language Models (LLMs) for enterprise use. While LLMs show high functional correctness on standard benchmarks, they often produce code that is verbose, complex, insecure, and difficult to maintain. The presentation introduces an evaluation framework and a proposed agent-centric development cycle to address these shortcomings and ensure LLM-generated code meets engineering standards.

This talk explores the engineering challenges and architectural patterns for building high-quality, low-latency voice agents at scale. It highlights that voice is a natural interface for human-computer interaction, moving beyond customer service to more complex applications. The core of the discussion revolves around the dominant pipeline architecture, its components, and the critical trade-offs involved in achieving real-time, intelligent, and natural-sounding voice interactions.

This talk introduces the concept of spec-driven testing for AI agents, arguing that simply using larger models or extensive datasets is insufficient for ensuring agent safety and reliability. It proposes a more comprehensive approach to defining and testing agent behavior by specifying not just expected outputs but also the context, rules, domain knowledge, and robustness requirements relevant to the agent's intended task.

This talk outlines the maturity phases of running evaluations for AI agents, emphasizing that evals are crucial for ensuring agent quality, mitigating risks, and understanding performance improvements. The speaker suggests a progression from initial human-based assessments to more automated and complex evaluation strategies as agent complexity increases. The core idea is to systematically build confidence in agent behavior before and after deployment.

This talk addresses the challenges of AI evaluations, which are currently scattered, quickly become outdated, and often lack transparency and verifiability. The speakers propose solutions to democratize the evaluation process, enabling a broader community to contribute to and benefit from robust AI benchmarking. Their work aims to foster more equitable AI development by allowing diverse expertise to shape evaluation standards.

This talk explores how to move beyond traditional code coverage metrics to ensure true functionality testing, especially in the context of AI-assisted development. It argues that while AI can increase code output, its true productivity gains are amplified by clean codebases and robust testing. The presentation introduces Playwright as a tool for end-to-end browser testing, demonstrating how it can be integrated with AI agents to accelerate development workflows, particularly within a test-driven development (TDD) framework.

This talk focuses on the practical aspects of evaluating and improving AI agents, moving beyond simple "vibe checks" to implement robust testing frameworks. It emphasizes that effective evaluation is crucial for shipping reliable AI applications, especially agents, which are prone to cascading failures. The session introduces methods for capturing agent behavior through tracing, categorizing failures, and implementing various types of evaluations (code-based, LLM-as-judge, human) to ensure agents perform as expected and to drive iterative improvements.

This talk argues that traditional static testing methods are insufficient for evaluating adaptive AI systems. As AI applications become more dynamic and intent-driven, evaluation strategies must evolve to become equally malleable. The core thesis is that static benchmarks fail to capture the emergent behaviors and changing user interactions characteristic of modern AI, necessitating a shift towards more adaptive and continuous evaluation approaches.

This talk introduces Pydantic AI's agent optimization capabilities, focusing on the Jeppa library for prompt optimization and managed variables for dynamic configuration. The core idea is to systematically improve agent performance by iteratively refining prompts and configurations, demonstrated through a practical example of analyzing political family ties in MP data. The presentation highlights how these tools can lead to more accurate and efficient AI agents.

Building effective evaluation (eval) platforms for AI agents is a complex systems problem, not just a UI challenge. While starting with simple tools like spreadsheets is a valid first step, maturing requires moving towards more robust solutions that facilitate experimentation and integrate with production data. The core difficulty lies in managing the unique characteristics of AI agent traces, which are often large, unstructured, and high-velocity, demanding specialized data infrastructure.

This talk challenges the perception that AI models are rapidly approaching general intelligence, as suggested by steadily increasing benchmark scores. It argues that despite impressive progress, models still exhibit significant weaknesses, particularly in handling nonsensical or complex, real-world tasks. The presentation introduces a benchmark focused on nonsensical questions and analyzes user dissatisfaction data to reveal areas where models continue to struggle.

This talk introduces GEPA, a method for building calibrated LLM evaluators that align with human annotations. The core idea is to move beyond generic LLM judges, which often fail in production, by optimizing prompts using algorithms like GEPA. This approach aims to accelerate development cycles by providing reliable signals for both offline evaluations and online monitoring, ultimately contributing to the creation of a data flywheel for continuous AI improvement.

This talk introduces Spec-Driven Development as a method to enhance AI agent development, focusing on improving control, code quality, and reliability. It proposes a structured workflow that compresses the software development lifecycle, moving from prompt to requirements, design, and execution, with an emphasis on creating reproducible results. The approach aims to integrate established software engineering practices with AI capabilities to tackle more complex problems.

This talk introduces an evaluation framework for AI product managers (PMs) focused on shipping reliable AI applications. It emphasizes the critical need for robust evaluation, analogous to software testing but adapted for the non-deterministic nature of AI models. The framework aims to provide PMs with tools and methodologies to build confidence in AI product performance, moving beyond subjective "vibe coding" to data-driven "thrive coding."

This talk explores the evolution of evaluating AI models for coding tasks, from simple code snippets to complex codebases. It highlights challenges like data contamination and brittle test suites, proposing dynamic evaluation sets and LLM-based judges to ensure reliable and relevant assessments as AI capabilities advance. The discussion covers various stages of coding evaluation, emphasizing the need for benchmarks that reflect real-world performance and adapt to the rapid progress in AI.

This talk addresses the gap between the hype surrounding AI code generation and the reality of its impact on software quality. While AI tools can significantly boost productivity in writing code, they introduce new challenges related to quality assurance, security, and maintainability. The core thesis is that achieving substantial, reliable gains requires moving beyond basic code generation to implement robust, agentic quality workflows throughout the entire software development lifecycle.

This talk emphasizes the critical role of robust evaluation (evals) in the development and deployment of AI systems. It argues that evals should be actively engineered, not passively accepted, and that they are essential for both playing offense by identifying new use cases and defense by ensuring quality. The core thesis is that by treating evals as a continuous engineering process, organizations can better adapt to rapid model advancements and ship higher-quality AI products.

This talk explores the limitations of current AI evaluation metrics, particularly in generative media, by highlighting how they often fail to account for human perception and aesthetic judgment. The speaker argues that traditional metrics, like FID scores, can be misled by factors such as compression artifacts, leading to inaccurate assessments of AI model quality. The core thesis is that AI evaluation needs to evolve to incorporate human perceptual nuances and subjective qualities, moving beyond easily quantifiable but potentially superficial measures.

This talk discusses the evolution of AI model evaluation (evals), highlighting the shift from manual processes to automated optimization. It introduces Loop, an agent designed to enhance prompts, datasets, and scorers, thereby revolutionizing the eval process. The core thesis is that frontier models, particularly recent advancements, are now capable of significantly improving AI development workflows.

This talk introduces the concept of evals at the application layer, distinguishing them from traditional unit tests. It emphasizes that Large Language Models (LLMs) can be unreliable, leading to unexpected failures in AI applications even when basic functionality appears to work. The core thesis is that robust evals are crucial for building reliable AI products by systematically testing and measuring performance across a spectrum of user-driven scenarios.

The core thesis of this talk is that 2025 will finally be the year for robust AI evaluation, driven by the convergence of several key factors. The increasing understanding of AI by business leaders, coupled with budget shifts towards generative AI projects, has set the stage. Crucially, the rise of agentic systems, which make decisions and take actions, introduces significant complexity and risk, making rigorous evaluation a necessity rather than an option.

This talk introduces a practical framework for evaluating AI search systems, particularly those operating in dynamic, real-world environments. It highlights the limitations of traditional static benchmarks and proposes dynamic datasets and reference-free metrics as crucial components for assessing the performance and reliability of augmented AI systems. The core thesis is that continuous improvement in AI systems requires robust evaluation methods that adapt to the evolving nature of information and user interactions.

This talk addresses the critical need for robust evaluation and benchmarking strategies when deploying large language models (LLMs) into production. It highlights the inherent complexities and potential pitfalls of generative AI, emphasizing that scalability, reliability, and safety are paramount. The presentation introduces practical tools and methods to assess LLM performance, ensuring that models meet enterprise-level requirements before and during deployment.

This talk explores the fundamentals of information retrieval, focusing on the "R" in Retrieval Augmented Generation (RAG). It delves into both traditional keyword search and modern vector search, explaining their underlying mechanisms, strengths, and limitations. The presentation emphasizes that effective retrieval is crucial for accurate and relevant results in AI applications.

This talk addresses the gap between AI-generated code prototypes and production-ready software. It introduces Sculptor, an experimental coding agent environment designed to build trust in AI-generated code by focusing on identifying and preventing defects. The core thesis is that AI should be leveraged not just for code generation but also for ensuring its quality and reliability.

This talk focuses on building AI applications more quickly and safely, emphasizing the importance of type safety in refactoring and avoiding bugs. It introduces techniques for developing AI applications, particularly highlighting the use of Pydantic AI for extracting structured data and managing agentic loops. The discussion also touches upon the challenges of determining when an agent loop should terminate and the benefits of using validation errors to guide model retries.

This talk emphasizes the critical role of evaluations (evals) in developing robust AI applications, moving beyond simple accuracy or similarity metrics. It highlights that effective evals are fundamental for aligning applications with system goals, ensuring continuous improvement, and building user trust. The core thesis is that a strategic, data-driven approach to evals, tailored to specific use cases, is essential for scaling AI development.

This talk introduces HybridRAG, a system that combines graph and vector retrieval for enhanced information retrieval. It highlights the advantages of knowledge graphs in capturing detailed relationships between entities, offering a more comprehensive view than purely semantic approaches. The system is broken down into data processing, graph and vector database creation, and inferencing, with a focus on optimizing retrieval strategies and evaluating performance.

This talk introduces ARC-AGI-3, a new benchmark designed to measure artificial general intelligence (AGI) by focusing on interactive reasoning and skill acquisition efficiency. The benchmark aims to create problems that are solvable by humans but challenging for current AI, thereby guiding AI research and development towards human-level intelligence. It moves beyond single-turn, static benchmarks to simulate more realistic, open-world exploration and learning scenarios.

This talk argues that AI benchmarks function as memes, spreading ideas and shaping the development of AI models. As benchmarks become popular, models are trained and tested on them until saturation occurs. This cycle presents an opportunity for individuals to create new benchmarks that can influence the future direction of AI development, emphasizing the importance of human-defined goals and values in the process.

This talk focuses on mastering AI evaluation, moving from initial development to production. It emphasizes that even the best large language models (LLMs) require a robust testing framework due to issues like hallucinations and performance degradation with changes. Effective evaluation helps answer critical questions about model selection, cost-effectiveness, brand consistency, and ongoing improvement, ultimately reducing development time, costs, and enabling faster iteration.

Building effective AI agents and the platforms that enable non-technical users to create them is a complex challenge due to the inherent nondeterminism of AI and unpredictable user behavior. The process requires a shift from traditional software development to building a data flywheel that continuously collects feedback, understands usage patterns, and identifies failures to improve the product iteratively. This involves robust instrumentation, strategic feedback collection, and a tiered approach to evaluation.

This talk introduces the concept and practical application of evals for AI systems, emphasizing their role in ensuring quality, reliability, and correctness. Evals provide a structured testing framework to move beyond the non-deterministic nature of LLMs, enabling developers to rigorously build and iterate on AI applications. The presentation highlights how evals facilitate a feedback loop between offline development and online production monitoring, ultimately leading to improved AI products.

This talk argues that evals are crucial for the success and iteration of AI products, moving beyond simple unit tests for AI. Evals provide a simulated laboratory environment, allowing developers to test and refine models extensively before deploying to production. This process significantly speeds up development cycles and increases confidence in shipping AI features.

This talk focuses on building scalable LLM evaluation pipelines. It emphasizes that effective evaluation is crucial for developing high-quality AI products and goes beyond simple "LLM as a judge" approaches. The core thesis is that a comprehensive evaluation strategy involves multiple methods, continuous tuning, and integration into the development lifecycle to accelerate iteration and improve AI system performance.

This talk addresses the common pitfalls and deceptive practices in vector search benchmarking, often referred to as "benchmarketing." The core thesis is that many published benchmarks are misleading due to biased scenario selection, outdated competitor versions, and a lack of focus on crucial factors like data freshness and query relevance. The speaker emphasizes that truly meaningful benchmarks require rigorous, reproducible, and self-conducted evaluations tailored to specific use cases.

This talk addresses the challenge of ensuring AI agents function reliably by introducing observability-driven evaluation. It highlights that AI's non-deterministic nature makes traditional testing methods insufficient. The core thesis is that by using AI itself to evaluate AI outputs, developers can gain crucial insights into agent performance, identify failures at granular levels, and implement targeted improvements.

This talk focuses on the critical process of evaluating AI agents, emphasizing a shift from ad-hoc testing to a more methodical approach. It highlights that effective evaluation should begin at the earliest stages of AI development, not as an afterthought. The presentation introduces tools and frameworks, particularly the Azure AI Evaluation SDK, to ensure AI agents behave correctly and safely as they gain more independence.

This talk explores the evolution and future of AI-powered search and retrieval, moving beyond traditional keyword matching to understand user intent and conceptual relationships. It highlights how AI search systems can provide more grounded and relevant responses, particularly in applications like Retrieval-Augmented Generation (RAG), by leveraging embeddings and advanced techniques. The discussion also touches upon the increasing importance of multimodality and agentic capabilities in shaping the next generation of search technologies.

This talk explores how graph data science can enhance reasoning and retrieval in AI applications, particularly within the context of Retrieval Augmented Generation (RAG). It emphasizes that graphs provide a way to understand data relationships beyond simple vector similarity, enabling more comprehensive and context-aware responses. The session introduces graph concepts and demonstrates how to leverage graph analytics to manage and improve data quality for AI systems.

This talk introduces GraphRAG, a method for optimizing Large Language Model (LLM) context windows using graph structures for retrieval. The core thesis is that LLM memory with structure is a key enabler for building effective AI applications, and when paired with agents, this combination offers even greater power. The presentation demonstrates GraphRAG's application in code understanding and feature development, alongside the announcement of Benchmark QED, an open-source tool for evaluating LLM systems.

This talk introduces Agentic GraphRAG, a method for simplifying data retrieval across both structured and unstructured sources. It proposes using a knowledge graph as an intermediary layer to enhance agentic workflows. By modeling data within a graph, agents can more accurately decompose complex questions, pull relevant information, and perform analytical tasks that go beyond simple semantic search.

This talk explores the challenges and strategies for testing AI-enabled features, particularly agentic editing in code editors. It emphasizes the shift from deterministic testing to embracing stochastic evaluations when LLMs are involved. The core thesis is that rigorous, empirical software development practices, adapted for the non-deterministic nature of AI, are crucial for building reliable AI-powered products.

This talk emphasizes that robust evaluations are the most critical, yet often overlooked, component for successfully scaling generative AI workloads. The speaker argues that evaluations are not merely for measuring quality but are primarily a tool for discovering problems within AI systems. Implementing a strong evaluation framework is presented as the key differentiator between a project that remains a science experiment and one that achieves production-level success and scalability.

This talk addresses the shortcomings of current Retrieval Augmented Generation (RAG) evaluation methods, arguing that they are fundamentally broken. The presenters contend that most benchmarks rely on simple, local questions with easily identifiable answers within specific text chunks, which does not reflect real-world data complexity. This leads to a cycle of optimizing for flawed benchmarks, resulting in RAG systems that perform poorly when deployed with actual user data.

This talk explores how the Model Contest Protocol (MCP) can be used to stabilize AI agents and agent networks. The core idea is that by systematically evaluating agent behavior and providing feedback, agents can learn to improve their performance and consistency, especially when tackling complex problems. This approach aims to create more controllable, transparent, and self-correcting agent systems.

This talk argues that current AI benchmarks are fundamentally flawed and often manipulated, leading to misleading performance claims. The speaker contends that the immense financial and market value tied to benchmark scores incentivizes companies to game the system through various deceptive practices. Ultimately, the presentation advocates for building custom, use-case-specific evaluations rather than relying on public benchmarks.

This talk introduces open-rag-eval, an open-source project designed for scalable Retrieval-Augmented Generation (RAG) evaluation. It addresses the common challenge of RAG evaluation requiring "golden answers" or "golden chunks," which is often impractical and non-scalable. The project, developed in collaboration with the University of Waterloo, offers a research-backed approach to evaluate RAG pipelines without relying on pre-defined correct answers.

This talk addresses the critical need for robust evaluation frameworks for AI agents, moving beyond initial development to production readiness. It emphasizes that as AI agents become more sophisticated, particularly with multimodal and voice capabilities, ensuring their reliability and effectiveness in real-world scenarios requires a structured approach to testing and validation. The core thesis is that comprehensive evaluation at every stage of an agent's operation is essential for scaling success and building trust.

This talk explores the necessity of domain-specific language models in finance, challenging the notion that general models with high accuracy are sufficient. The evaluation presented reveals that while general models perform well on standard benchmarks, they struggle with real-world complexities like misspellings, incomplete queries, and crucially, grounding answers within provided context. This suggests a significant gap in robustness and reliability, indicating that specialized models and robust system design are still essential for critical applications.

This talk addresses the current limitations and challenges in building and evaluating AI agents. While agents are increasingly integrated into products, ambitious visions of their capabilities are far from realized. The core thesis is that AI engineering must prioritize reliability and rigorous evaluation, treating these as first-class concerns to overcome the inherent stochasticity of language models and move beyond misleading benchmarks.

This talk introduces a framework for evaluating AI agents, dividing agent evaluation into semantic and behavioral aspects. Semantic evaluation focuses on how an agent's internal representations align with reality, while behavioral evaluation assesses how an agent's actions and tool usage contribute to achieving its goals. Both aspects are further categorized into single-turn and multi-turn scenarios, providing a comprehensive map for understanding and measuring agent performance.

This talk outlines the challenges and solutions for building scalable evaluation systems for AI, particularly in mission-critical domains like healthcare where errors have significant consequences. It emphasizes the limitations of traditional human review and offline datasets, advocating for real-time, reference-free evaluation methods to ensure customer trust and enable rapid response to issues.

This talk argues that traditional evaluation methods for AI systems are often insufficient and can lead to meaningless results. The core thesis is that evaluations must be dynamic and continuously aligned with real-world usage and user expectations, rather than relying on static, generalized criteria.

This talk focuses on optimizing Large Language Model (LLM) quality through fine-tuning, addressing common pain points like high operational costs and the inability to meet production-ready quality standards. It positions fine-tuning within a broader "crawl, walk, run" strategy for LLM quality, emphasizing that it should be considered after prompt engineering and Retrieval Augmented Generation (RAG) have been explored. The presentation outlines a continuous deployment cycle for fine-tuned LLMs, including data collection, model fine-tuning, deployment, and evaluation, aiming to demystify the process for AI engineers.

This talk demonstrates how to build, evaluate, and deploy a Retrieval-Augmented Generation (RAG) based copilot for a retail environment using Azure AI services. The core concept involves creating a chatbot that can answer customer questions by accessing product information from a vector database and customer data from a relational database, all orchestrated through Azure AI Studio's Prompt Flow.

This talk focuses on the practical challenges and solutions for building effective LLM evaluation systems in real-world applications. It distinguishes between model evals, which rank models against benchmarks, and task evals, which assess whether an LLM application is functioning correctly for its intended purpose. The core thesis is that robust task evals, especially those providing explanations for failures, are crucial for iterating and improving deployed LLM applications.

This talk addresses the challenge of evaluating language models for specific use cases, highlighting that generic metrics and benchmarks often fall short. It proposes a method where another language model is used to grade the output of the model being evaluated, allowing for the creation of specialized metrics tailored to an application's needs. This approach enables data-driven decisions when selecting LLMs.

This talk introduces llmeval, a command-line tool designed to help teams ship reliable large language model (LLM) applications. It addresses the challenge of defining and measuring "good" in generative AI, which is crucial for evolving applications, changing prompts, or considering different models. llmeval provides a structured approach to testing and evaluating LLM outputs, enabling more confident development and deployment.

This talk introduces BotDojo, an AI enablement company focused on helping businesses deploy AI applications to production. The demonstration highlights how to use synthetic data generation combined with evaluations to improve the performance of chatbots. It showcases a low-code editor for building AI flows, emphasizing the importance of detailed tracing for debugging and the integration of evaluations to identify issues like hallucinations and insufficient information retrieval.

This talk addresses the critical and evolving landscape of AI evaluations, particularly as systems become more agentic and automated. It argues that current evaluation methods are often simplistic and prone to "benchmark hacking," failing to capture the true performance and values embedded in AI systems. The discussion emphasizes the need for more robust, multifaceted evaluation strategies that consider real-world scenarios, user experience, and the underlying values driving AI development to ensure these systems align with human goals.

This talk addresses the challenge of measuring the return on investment (ROI) of AI tools, particularly for developers. The core thesis is that while AI can significantly enhance developer productivity, demonstrating its business impact requires moving beyond anecdotal evidence or simple engagement metrics to more structured evaluation frameworks. The speaker emphasizes that the difficulty in measuring AI ROI is akin to the NP-hard problem of measuring developer productivity itself, suggesting that a precise, universally applicable solution is unlikely, but practical frameworks can be developed.

This talk explores a "ghost pilot" system designed to enhance code quality and security within the CI/CD pipeline. Unlike quick, just-in-time coding assistants, this system performs deliberate, iterative analysis. It aims to automate tedious tasks like improving code comments, adding unit tests, identifying security vulnerabilities, and suggesting fixes, thereby freeing up developers to focus on core coding tasks.

This talk focuses on optimizing Retrieval Augmented Generation (RAG) systems by employing an evaluation-driven approach. It highlights the common challenges encountered in RAG implementations, from data processing and retrieval to response generation, and proposes systematic methods for improvement. The core thesis is that continuous, data-driven evaluation is essential for effectively refining RAG performance and achieving desired outcomes.

This talk focuses on architecting and testing reliable AI agents, moving beyond traditional chains and open-ended agents that suffer from poor reliability. The presenter introduces Lang Graph as a solution for building controllable agents that balance flexibility with robustness by expressing control flows as graphs with nodes and edges, incorporating state for memory.

This talk outlines a systematic approach to constructing domain-specific LLM evaluation systems. It emphasizes the importance of moving beyond initial "vibe checks" and prompt engineering to establish a robust framework for consistent AI improvement. The core thesis is that a well-defined evaluation system, built on foundational principles, is crucial for developing production-ready AI applications and unlocking advanced capabilities like fine-tuning.

This talk explores how Codeium addresses limitations in traditional retrieval methods for AI agents, particularly within code generation. The core thesis is that current embedding-based retrieval hits a ceiling due to its inability to effectively reason over multiple contextual elements simultaneously. Codeium proposes a vertically integrated approach, leveraging custom infrastructure and models to enable a more powerful and efficient retrieval system called M query.

This talk explores the limitations of basic retrieval-augmented generation (RAG) loops and argues for more sophisticated memory systems to power advanced AI applications. It highlights the need for RAG systems that can incorporate human feedback, self-update based on agent interactions, and dynamically adapt to evolving data. The core thesis is that future powerful AI applications will require retrieval systems far beyond simple search indexes.