World's Fair 2026

Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Abed Matini , Ogilvy

Overview

This talk introduces a framework for building chatbots that bypass the limitations and costs associated with multimodal inputs and complex tool integrations. It proposes a hybrid Retrieval Augmented Generation (RAG) approach using SQL, Reciprocal Rank Fusion (RRF), and live telemetry to improve efficiency and accuracy. The core idea is to optimize data ingestion and retrieval through structured processing, enabling more effective querying of documents and reducing the computational burden on LLMs.

Who should watch

• AI engineers and developers building RAG systems.
• Product Managers looking to integrate LLMs into applications.
• Builders seeking to optimize document processing and querying for chatbots.
• Those interested in cost-effective LLM solutions and local model deployment.
• Engineers focused on observability and guardrails in AI applications.

Key takeaways

• **Hybrid RAG for Efficiency:** Combine semantic and keyword search (e.g., BM25) for more accurate and relevant retrieval, especially for structured data like product SKUs or medical information.
• **Structured Data Ingestion:** Convert documents to Markdown first using tools like DocLink, then chunk them based on strategies like headings or paragraphs for cleaner data. This avoids token waste and improves LLM comprehension.
• **Local Model Deployment:** Utilize local models (e.g., Ollama) and embedding models that can run on CPU, reducing costs and enabling self-hosting.
• **Python Agents for Specific Tasks:** Employ Python functions for tasks like retrieving the current date or performing calculations instead of relying on LLMs, which enhances speed and control.
• **Observability with LangFuse:** Integrate tools like LangFuse for tracking chat interactions, latency, and model performance, providing insights for debugging and improvement.
• **Guardrails and Prompt Injection Prevention:** Implement guardrails within the code, before sending queries to the LLM, to block irrelevant or malicious inputs, reducing hallucination and improving safety.
• **Chunking Strategies Matter:** Experiment with different chunking methods (heading-based, paragraph-based, fixed-size, sentence-based) to find the optimal way to represent document data for retrieval.
• **Smaller Models Can Suffice:** With effective data vetting and chunking, smaller LLMs can provide good results and reduce hallucination compared to larger, more resource-intensive models.

Notable quotes

Watch on YouTube →

Unofficial community note. Prefer the recording for nuance.