Europe 2026
Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft
Overview
This talk addresses the critical need for robust observability in AI agents, highlighting the gap between an agent's intended functionality and its actual behavior in production. The presenters introduce the "Mind the Gap" analogy to emphasize the importance of continuous evaluation and monitoring throughout an agent's lifecycle. They advocate for a proactive approach to identify and address discrepancies, ensuring agent reliability, quality, and safety.
Who should watch
- AI Engineers
- Product Managers
- Developers building and deploying AI agents
- Anyone concerned with agent reliability and safety
- Those looking to improve their agent development and debugging workflows
Key takeaways
- Agent observability is crucial for bridging the gap between intended and actual agent behavior, especially as agents are non-deterministic.
- A three-pronged approach to agent reliability is essential: evaluation (performance, quality, safety), monitoring (continuous over time), and optimization (acting on data).
- Microsoft Foundry provides a platform for building, hosting, observing, and managing agents, with observability integrated across early development, production debugging, and multi-agent system management.
- Tracing, built on OpenTelemetry, allows for detailed monitoring of agent actions, tool calls, and message exchanges, enabling better debugging.
- Evaluations, including built-in and custom options, are vital for assessing agent output against quality, risk, and safety metrics at various stages of the workflow.
- The concept of "red teaming" involves proactively testing agents for vulnerabilities by simulating adversarial attacks to identify potential security and safety loopholes.
- Newer approaches, like the "Observe Skill" within Microsoft Foundry, leverage coding agents to automate the observability loop, including generating evaluation datasets, optimizing prompts, and managing agent versions with human oversight.
- Integrating AI telemetry with broader cloud monitoring solutions like Azure Monitor provides a holistic view of system performance and potential impacts.
Notable quotes
*Agents are non-deterministic. That's not just a problem for demos; that's also a problem for real life when you actually get to production.*
*The gap in agent observability is three things: agent drift from requirements, the need to accelerate the eval-optimize loop, and closing the loop on too many models and no existing data.*
Unofficial community note. Prefer the recording for nuance.