Europe 2026
The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks
Overview
This talk outlines a production AI playbook, a framework designed to guide the deployment of AI systems at an enterprise scale. It addresses common pitfalls encountered when moving AI from demos to production, emphasizing the need for structured approaches to evaluation, observability, data foundation, orchestration, and governance. The framework aims to ensure AI systems are measurable, accountable, and reliable in real-world applications.
Who should watch
- AI Engineers
- Product Managers
- Builders working on deploying AI systems
- Those facing challenges in scaling AI from demos to production
- Teams needing to ensure AI reliability and accountability
Key takeaways
- A common pattern of AI development involves choosing a model, building demos in controlled environments, and then facing issues when deploying to production due to a lack of observability, evaluation, and governance.
- The Production AI Playbook consists of five pillars: Evaluation, Observability, Data Foundation, Orchestration, and Governance, which should be considered before starting a project and built gradually.
- Evaluation involves defining success metrics with numbers, building test cases from real-world scenarios, and automating testing pipelines, with layers including deterministic checks, LLM judges, and behavioral analysis.
- Observability, or tracing, is crucial for understanding every decision an AI agent makes, enabling debugging, dispute resolution, and regulatory compliance.
- A robust Data Foundation is essential, distinguishing between question data (for AI responses) and tracking data (for observability), requiring a clear strategy for collection and serving.
- Orchestration becomes critical with multiple agents, requiring patterns like orchestrator-worker or choreography to manage complexity and inter-agent communication.
- Governance ensures accountability, audit trails, pre-validation of personal information, and robust change management for prompts and models.
- A case study demonstrated that prioritizing evaluation and data foundation before model selection significantly accelerated successful production deployment.
Notable quotes
*If we can't see what it is actually doing, if we can't trace every decision that it's making, it's no use in production.*
*Agents don't forgive you. Agents will go, find it wrong, they'll give you the wrong answer confidently.*
*Treat prompt versioning as change management in enterprise grade solution.*
Unofficial community note. Prefer the recording for nuance.