World's Fair 2025
Production software keeps breaking and it will only get worse — Anish Agarwal, Traversal.ai
Overview
The talk argues that as AI tools increasingly automate code development, the complexity of system design and production troubleshooting will grow, potentially leading engineers to spend more time on on-call duties. The core thesis is that current approaches to AI-assisted troubleshooting are insufficient and a new, more integrated method is needed to handle the escalating complexity of production incidents.
Who should watch
- Engineers and Product Managers involved in shipping and maintaining production software.
- Those concerned about the increasing complexity of debugging and incident response.
- Individuals exploring the application of AI agents and causal inference in software engineering.
- Teams looking to improve their Mean Time To Resolution (MTTR) for production incidents.
Key takeaways
- AI's increasing role in code development may reduce time spent on coding but could exacerbate challenges in system design and troubleshooting.
- Traditional AI ops and current LLM approaches struggle with the scale and complexity of production data, leading to false positives or an inability to process vast log volumes.
- Agent-based systems relying on static runbooks are prone to becoming outdated quickly, and broad system searches can be prohibitively slow.
- A novel approach combining causal machine learning (statistics), advanced reasoning models (semantics), and a swarm of agents for control flow is proposed to automate troubleshooting.
- This integrated approach aims to provide autonomous, first-principles debugging for novel incidents by identifying cause-and-effect relationships and semantic context within telemetry data.
- A case study with Digital Ocean demonstrated a significant reduction in Mean Time To Resolution (MTTR) by using an AI system to sift through vast amounts of observability data and pinpoint root causes.
- The solution involves processing trillions of logs and metrics across various observability tools, highlighting the need for robust AI infrastructure alongside agent capabilities.
- The strategies of exhaustive search and agent swarms can be applied to other domains like network observability and cybersecurity.
Notable quotes
*Production software keeps breaking and it will only get worse.*
*The problem of troubleshooting. How do you automate that part of it?*
*We're talking trillions of logs. So yes, it's an AI agents problem and I I like thinking about the AI agents problem, but in reality this is just as much an AI infrastructure problem.*
Unofficial community note. Prefer the recording for nuance.