User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch

The State of AI Engineering 2026 survey highlights evaluation (evals) as the most significant challenge across AI stacks. This talk delves into the survey results, emphasizing the critical need for robust evaluation methodologies in AI development. Addressing evals is presented as paramount for building reliable and high-quality AI systems.

This talk focuses on the critical importance of building effective evaluation systems for AI models, moving beyond superficial metrics to create benchmarks that genuinely reflect real-world performance and user needs. The presenters emphasize that robust evals are essential for shipping reliable AI products and driving meaningful improvements in model quality.

This talk addresses the common problem of AI evaluations that fail to predict real-world performance. The core thesis is that offline evaluations often use simplistic "customers" and test sets that don't reflect the complexity and adversarial nature of actual user interactions, leading to shipped models that fail in production. A solution involves building more realistic, adversarial user simulators trained on real data.

This talk emphasizes the critical need for rigorous evaluation of AI skills before deployment, arguing that shipping skills without proper testing can lead to unpredictable behavior and degraded performance. The speaker highlights the distinction between agents developers use for personal productivity and agents built for consumers, noting that end-users lack the context to troubleshoot skill invocation issues. The core thesis is that comprehensive evaluations are essential for ensuring skill reliability, managing costs, and determining when skills can be retired as models improve.

This talk addresses the significant gap between the rapidly advancing reasoning capabilities of large language models (LLMs) and the slower evolution of retrieval systems. The core thesis is that LLMs are bottlenecked not by their reasoning power, but by their ability to access the correct knowledge. By improving retrieval tools and teaching agents to use them effectively, most of this knowledge gap can be closed, unlocking LLMs for complex tasks beyond coding.