Europe 2026
How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust
Overview
This talk differentiates traditional observability from agent observability, arguing that agents' non-deterministic nature and complex data structures necessitate a distinct approach. Traditional observability focuses on uptime and technical performance metrics like latency and error rates, using established tools. Agent observability, however, must also account for qualitative aspects of agent performance, such as grounding, tool usage, and adherence to brand standards, which require handling vast amounts of semi-structured and unstructured data.
Who should watch
- AI Engineers
- Product Managers
- Builders of AI agents
- Those struggling to bring AI proofs-of-concept to production
- Teams needing to understand agent behavior beyond simple uptime metrics
Key takeaways
- Traditional observability focuses on deterministic applications and metrics like uptime, latency, and error rates, often using tools like Grafana or Datadog.
- Agent observability must address non-deterministic behavior, requiring measurement of qualitative aspects like response grounding, tool utilization, and brand alignment.
- Agent traces are significantly more complex, being semi-structured, voluminous (potentially gigabytes), and containing unstructured text, posing a unique systems challenge for ingestion and processing.
- A specialized database approach is needed for agent traces, incorporating features like write-ahead logs for instant visibility and full-text indexing (e.g., using frameworks like Tantivy) to query unstructured data.
- Agent observability involves a broader range of users, including non-technical subject matter experts, who can provide valuable qualitative feedback and insights.
- Observability and evals for agents are closely related, with evals being batch runs of known inputs, while observability handles real-time, unknown inputs.
- Future directions include using lightweight LLMs on agent traces for embedding, clustering, and topic modeling to understand user intent, sentiment, and identify issues.
- Human annotation plays a crucial role in agent observability, allowing experts to grade agent performance and justify their assessments, which can then inform automated scoring functions.
Notable quotes
*Traditional observability is all about uptime and technical performance.*
*Agent applications are very much non-deterministic. We're curious about why an agent might take one path versus the other.*
*Agent traces are really nasty. They're in a variety of different ways.*
Unofficial community note. Prefer the recording for nuance.