Session brief

Evaluating Domain Specific LLMs for Real World Finance — Waseem Alshikh, Writer

Waseem Alshikh , Writer

Overview

This talk explores the necessity of domain-specific language models in finance, challenging the notion that general models with high accuracy are sufficient. The evaluation presented reveals that while general models perform well on standard benchmarks, they struggle with real-world complexities like misspellings, incomplete queries, and crucially, grounding answers within provided context. This suggests a significant gap in robustness and reliability, indicating that specialized models and robust system design are still essential for critical applications.

Who should watch

• AI Engineers building or evaluating LLMs for specific industries.
• Product Managers assessing the suitability of general vs. domain-specific models for their products.
• Builders seeking to understand the limitations of current LLM technology in real-world scenarios.
• Anyone concerned with LLM hallucination and grounding accuracy in high-stakes domains like finance.

Key takeaways

• General LLMs achieve high accuracy on standard benchmarks but falter when faced with real-world data imperfections such as misspellings, incomplete queries, or out-of-domain inputs.
• A significant gap exists between a model's ability to generate an answer and its capacity to ground that answer accurately within the provided context.
• Larger, more advanced "thinking" models often perform worse on context grounding compared to smaller, domain-specific models, leading to higher hallucination rates.
• The evaluation introduced categories like query failure (misspellings, incomplete, out-of-domain) and context failure (missing context, OCR errors, irrelevant context).
• Even the best-performing models in the evaluation did not exceed 81% in robustness and context grounding, implying that a substantial portion of requests may still yield incorrect results.
• The data suggests that for reliable deployment, a full-stack approach including robust systems, guardrails, and grounding mechanisms is necessary, not just the LLM itself.
• The conclusion is that building and continuing to develop domain-specific models remains crucial, as current general models lack the necessary context-following capabilities for critical applications.
• The evaluation dataset, methodology, and leaderboard are available as open-source resources on GitHub and Hugging Face.

Notable quotes

Watch on YouTube →

Unofficial community note. Prefer the recording for nuance.