CI in the Era of AI: From Unit Tests to Stochastic Evals — Nathan Sobo, Zed
This talk explores the challenges and strategies for testing AI-enabled features, particularly agentic editing in code editors. It emphasizes the shift from deterministic testing to embracing stochastic evaluations when LLMs are involved. The core thesis is that rigorous, empirical software development practices, adapted for the non-deterministic nature of AI, are crucial for building reliable AI-powered products.
World's Fair 2025 15 min