Session brief
Fixing bugs in Gemma, Llama, & Phi 3: Daniel Han
Overview
This talk addresses common bugs and issues encountered when fine-tuning and deploying open-source large language models, specifically focusing on Gemma, Llama 3, and Phi 3. It provides practical solutions and best practices to ensure successful model training and inference, highlighting the importance of careful attention to tokenization, model templates, and export formats.
Who should watch
- AI Engineers
- Machine Learning Engineers
- Product Managers
- Developers working with open-source LLMs
- Anyone experiencing issues with fine-tuning or deploying models like Llama 3, Gemma, or Phi 3
- Users encountering NaN gradients or infinite generations during fine-tuning
Key takeaways
- Avoid using double BOS (Beginning Of Sentence) tokens during fine-tuning, as this can significantly degrade inference accuracy. Tools like unsoft can automatically detect and remove extra BOS tokens.
- When fine-tuning the Llama 3 base model, be cautious of untrained tokens (0-250, EOT, SOH, EOH) which can cause NaN gradients. These tokens should ideally be set to the mean of trained tokens or trained themselves. The Llama 3 instruct chat template should not be used with the base model.
- Ensure the pad token and EOS (End Of Sentence) token are not the same. If they are, it can lead to infinite generations because the EOS token might be masked out during loss calculation.
- Exporting fine-tuned models to formats like GGUF for llama.cpp requires careful attention to chat template consistency. Tools can now automatically generate the necessary model files for export.
- Community contributions have identified issues such as CPU-only conversion for GGUF/llama.cpp and the benefit of adding system prompts during fine-tuning for improved results.
- For longer context fine-tuning, techniques like offloading gradient checkpointing to system RAM can be effective without drastically increasing completion time, provided non-blocking calls are used.
- When preparing data for fine-tuning, especially with CSV files, columns may need to be merged, and chat templates must be carefully constructed, often requiring two iterations to handle dangling new lines.
Notable quotes
*Please check before you fine tune if you're using double BOS tokens.*
*The Llama 3 chat template will not work for the base model.*
*The pad token and the EOS token must not be the same.*
Unofficial community note. Prefer the recording for nuance.