World's Fair 2025
How agents broke app-level infrastructure - Evan Boyle
Overview
This talk addresses the challenges of building reliable AI applications, particularly focusing on the compute layer. Traditional web infrastructure is ill-suited for the long-running, non-deterministic nature of AI workflows, which often involve extensive data ingestion and complex agentic processes. The core thesis is that current infrastructure limitations lead to unreliable user experiences and hinder rapid experimentation, necessitating new architectural approaches.
Who should watch
- AI Engineers building agentic workflows
- Product Managers evaluating AI product feasibility
- Developers facing infrastructure limitations with long-running AI tasks
- Anyone interested in the practical challenges of shipping AI applications
Key takeaways
- Traditional web infrastructure, designed for low-latency requests, is inadequate for AI applications that can take seconds or minutes to complete.
- AI applications often require extensive data ingestion and processing, pushing developers into data engineering roles and highlighting the need for robust background job management.
- Existing serverless platforms typically have timeouts (e.g., 5 minutes) and lack native streaming support, making them unsuitable for multi-minute AI workflows.
- User experience suffers due to potential outages, rate limits, and the inability to resume or track progress on long-running tasks, leading to user frustration and increased drop-off rates.
- A separation of API and compute layers, combined with a message queue like Redis streams, can enable resumability and transparent error handling for background AI processes.
- An open-source library inspired by React's component model is proposed, focusing on reusable, independently testable components and workflows for building agentic applications.
- Key architectural principles include planning for long-running workflows from the start, separating compute and API planes, and leveraging streams for resumability.
- Shipping reliable AI applications involves careful attention to details like worker draining during deployments and robust error handling.
Notable quotes
*The infrastructure that was designed for the last 10 years of the web is not really designed for the applications that we're building today.*
*LM applications are really built on shoddy foundations. It's really hard to build reliable apps today.*
*The future is giving your agent instructions and letting it go off and do work for you and communicate with you when you need it.*
Unofficial community note. Prefer the recording for nuance.