World's Fair 2025
How Intuit uses LLMs to explain taxes to millions of taxpayers - Jaspreet Singh, Intuit
Overview
Intuit leverages Large Language Models (LLMs) within TurboTax to enhance taxpayer understanding of their tax situations and potential deductions. The company has developed a proprietary generative OS called Geno, designed for large-scale, secure, and reliable use in the highly regulated tax industry. This platform integrates various components, including UI elements and an orchestrator, to manage different LLM solutions and deliver a cohesive user experience.
Who should watch
- AI engineers and builders working on large-scale LLM applications.
- Product Managers and engineers focused on integrating AI into regulated industries.
- Teams concerned with LLM evaluation, accuracy, and latency.
- Developers exploring the use of LLMs for complex question answering and explanation generation.
Key takeaways
- Intuit processes millions of tax returns annually, necessitating scalable LLM solutions focused on accuracy and taxpayer confidence.
- The Geno platform, Intuit's generative OS, is built with a strong emphasis on safety, security, and regulatory compliance, addressing use cases not met by off-the-shelf tooling.
- For tax explanation, Intuit utilizes Anthropic's Claude models, with specific use cases employing static prompts for known queries and dynamic prompts for user-specific questions, often leveraging GPT-4 for the latter.
- Retrieval-Augmented Generation (RAG) and Graph RAG are employed to access and utilize Intuit's proprietary tax information, enhancing the personalization and accuracy of responses.
- A phased evaluation system is critical, involving manual reviews by tax experts for initial development and automated evaluations for ongoing iteration and monitoring, with LLM-as-a-judge being a key component of the automated process.
- Fine-tuning LLMs, such as Claude 3 Haiku, was piloted to improve response quality and reduce latency, though it proved highly specialized.
- Latency is a significant challenge, especially during peak tax season, requiring careful product design and fallback mechanisms to ensure a seamless user experience.
- *Evals are a must to launch. Focus on evals.* Clear guidelines and golden datasets are essential for effective evaluation.
Notable quotes
The core goal is to help users understand their taxes better and ensure they are getting the best deductions.
*Evals are a must to launch. Focus on evals.*
We are not having LLMs do the calculations at all. We're basically using the ground truth that is already existing in our systems as the numbers that we see.
Unofficial community note. Prefer the recording for nuance.