David Karam

This talk provides a practical framework for improving Retrieval Augmented Generation (RAG) systems by systematically layering techniques based on their complexity and impact. It emphasizes a quality engineering approach, starting with defined outcomes and product problems, then analyzing failures to select appropriate RAG techniques. The core thesis is that understanding where and why a system fails is crucial for making informed decisions about which RAG enhancements to implement, rather than getting lost in hype or theoretical debates.

This workshop focuses on building effective evaluation metrics for AI systems, moving beyond basic testing to create robust scoring systems. It emphasizes that evaluations are not just for testing but are the primary place where domain knowledge resides, enabling significant improvements in AI development. The session introduces a methodology for creating nuanced, calibrated metrics that correlate with desired outcomes, ultimately simplifying the AI development stack.