Europe 2026

LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

Dat Ngo

Overview

This talk introduces a platform for LLM observability, evaluation, and experimentation, emphasizing that building AI systems is fundamentally an engineering discipline. It highlights the importance of understanding what an AI system is doing (observability), how to measure its performance (evaluation), and how to systematically improve it (experimentation). The core thesis is that these processes, while complex in the non-deterministic world of AI, can be managed and eventually automated through robust engineering practices and tooling.

Who should watch

• AI Engineers
• Product Managers
• Builders of AI agents and harnesses
• Those responsible for productionizing AI systems
• Individuals facing challenges in debugging and improving AI model performance
• Teams looking to implement systematic testing and quality assurance for AI applications

Key takeaways

• Observability in AI systems, similar to traditional software, involves tracking what's happening through mechanisms like traces and spans, with OpenTelemetry being a key pattern.
• Evaluation of AI systems can take many forms, including LLM-as-a-judge, human feedback, golden datasets, and deterministic logic-based checks, all aimed at deriving meaningful signals.
• Improvements made to AI systems can inadvertently introduce regressions, making robust evaluation and experimentation platforms crucial for identifying and fixing these issues.
• The platform supports various levels of evaluation granularity, from single component (span) evals to multi-span and trajectory evals that assess entire workflows or conversational states.
• A key aspect of the platform is enabling both technical users and domain experts (like PMs) to define and run evaluations, bridging the gap between coding and understanding AI experience.
• The future of AI development involves automating observability, evaluation, and experimentation, with the goal of making the entire improvement flywheel feel seamless and magical.
• Arize offers two products: Arize Phoenix, an open-source, single-container solution for local deployment, and Arize AX, a comprehensive enterprise solution for large-scale deployments.
• The platform aims to automate the process of identifying issues, creating evaluations, and running experiments, allowing AI systems to self-monitor and improve.

Notable quotes

Watch on YouTube →

Unofficial community note. Prefer the recording for nuance.