World's Fair 2025
Layering every technique in RAG, one query at a time - David Karam, Pi Labs (fmr. Google Search)
Overview
This talk provides a practical framework for improving Retrieval Augmented Generation (RAG) systems by systematically layering techniques based on their complexity and impact. It emphasizes a quality engineering approach, starting with defined outcomes and product problems, then analyzing failures to select appropriate RAG techniques. The core thesis is that understanding where and why a system fails is crucial for making informed decisions about which RAG enhancements to implement, rather than getting lost in hype or theoretical debates.
Who should watch
- AI Engineers looking to improve RAG system performance.
- Product Managers aiming to understand the trade-offs in RAG development.
- Builders evaluating different techniques for information retrieval and generation.
- Those struggling to orient themselves within the vast landscape of RAG techniques and buzzwords.
- Teams needing to establish a systematic approach to quality engineering for AI systems.
Key takeaways
- Begin RAG system improvement by defining product outcomes and quality bars, then analyze query sets and identify specific failure points.
- Adopt a "complexity-adjusted impact" or "stay lazy" approach: address what's broken and avoid unnecessary complexity.
- Start with simpler techniques like in-memory retrieval (leveraging LLM context windows) and BM25 for keyword-based search before moving to more complex methods.
- Re-rankers (like cross-encoders) can improve relevance by attending to query and document simultaneously but are computationally expensive, requiring retrieval of a smaller set.
- Standard relevance metrics may not suffice for domain-specific or nuanced queries; consider custom embeddings to model specific vector spaces.
- User preference signals (clicks, ratings) are critical for ranking and often outperform pure semantic relevance, especially in e-commerce or personalized applications.
- Orchestration challenges, particularly with agents using tools, can lead to incorrect queries; consider fanning out complex queries into smaller, tailored ones.
- When systems become too complex or costly, distillation can be used to create smaller, specialized models that maintain quality for specific tasks.
Notable quotes
*Always start with outcomes. You're always trying to solve some product problem.*
*Complexity adjusted impact or you know stay lazy in a sense always look at what's broken and if it's not broken don't fix it.*
*Relevance is not ranking and this is something you know we learned in Google search.*
Unofficial community note. Prefer the recording for nuance.