Session brief
The Devops Engineer Who Never Sleeps — Diamond Bishop, Datadog
Overview
This talk explores Datadog's development of AI agents designed to assist with DevOps tasks, focusing on the "AI on-call engineer" and "AI software engineer." The core thesis is that as AI capabilities advance, platforms like Datadog can evolve from mere observability tools into intelligent agents that proactively manage systems, reduce human toil, and enhance developer productivity. The presentation highlights the challenges and learnings in building, evaluating, and integrating these agents into existing workflows.
Who should watch
- AI engineers and builders working on agent development.
- Product Managers and engineers focused on shipping AI-powered features.
- DevOps and SRE professionals seeking to automate incident response and system maintenance.
- Anyone interested in the practical application of AI agents in enterprise environments.
- Teams struggling with alert fatigue and manual debugging processes.
Key takeaways
- Datadog is developing AI agents, including an AI on-call engineer to handle alerts and an AI software engineer to identify and fix code errors proactively.
- The AI on-call engineer can autonomously investigate alerts by analyzing logs, metrics, and traces, referencing runbooks, and suggesting or executing remediations.
- The AI software engineer acts as an error-tracking assistant, proposing code fixes and generating tests to prevent future incidents.
- Building effective AI agents requires a strong focus on evaluation, with emphasis on defining clear tasks, using domain experts as design partners, and implementing robust offline, online, and living evaluation strategies.
- Team composition is crucial, favoring optimistic generalists skilled in coding and rapid iteration over a large number of ML experts.
- User experience (UX) for human-AI collaboration is vital, with a preference for agents that function like human teammates rather than just new interfaces.
- Observability remains critical, with Datadog developing specialized "LLM observability" tools to visualize complex agent workflows and debug issues effectively.
- The future likely involves agents becoming significant users of SaaS platforms, necessitating a shift in how products are designed and integrated.
Notable quotes
*The future is uncertain this kind of ambiguity creates opportunity but there's a lot of potential for us.*
*Domain experts eval eval eval I can't stress this enough.*
*The old ux patterns are changing be comfortable with that.*
Unofficial community note. Prefer the recording for nuance.