Session brief
Training Albatross An Expert Finance LLM: Leo Pekelis
Overview
This talk details the development of Albatross, an expert finance Large Language Model (LLM). It argues that generalist LLMs are insufficient for complex domains like finance, necessitating custom-trained models and extended context lengths. The approach involves creating a domain-specific finance LLM through automated data curation and a tailored training pipeline, alongside developing models with significantly increased context windows to mitigate hallucinations and enable more sophisticated in-context learning.
Who should watch
- AI engineers and builders working on domain-specific LLMs.
- Product Managers and engineers looking to improve LLM accuracy and reduce hallucinations in specialized applications.
- Researchers interested in LLM training methodologies, data curation, and context length extensions.
- Individuals exploring open-source LLM solutions for finance or other technical domains.
Key takeaways
- Generalist LLMs struggle with specialized domains like finance due to a lack of deep, relevant training data.
- Training a domain-specific LLM requires an automated data pipeline, including automated data curation using techniques like membership inference to filter unseen data.
- The training process involves continuous pre-training on curated data followed by alignment stages like supervised fine-tuning and preference optimization.
- Hallucinations in LLMs often stem from training data deficiencies, such as outdated information or inconsistencies.
- In-context learning is an effective method to reduce hallucinations by providing relevant information directly in the prompt at inference time.
- Context length is a significant bottleneck for in-context learning, limiting the amount of information that can be processed.
- Extending context length to around a million tokens enables the inclusion of thousands of examples for on-the-fly domain learning and allows models to perform complex reasoning by attending to all tokens.
- Gradient has open-sourced two models: V-Alpha-TR, a finance-specific LLM based on Llama 2, and a Llama 3-based model with a 1 million token context extension.
Notable quotes
*Large language models are not one-size-fits-all, especially when you get to more complicated use cases.*
*In context learning... is the most direct and in-sample efficient way to reduce hallucinations.*
*Long context language models give you more grounded and robust systems and there are fewer moving parts.*
Unofficial community note. Prefer the recording for nuance.