World's Fair 2024
The GenAI Maturity Curve or You Probably Don't Need Fine Tuning: Kyle Corbitt
Overview
This talk argues that most AI engineers and builders likely do not need fine-tuning for their current projects. The core thesis is that while fine-tuning can significantly improve model performance and cost-effectiveness, it introduces upfront time investment and reduces flexibility. The presentation aims to help the audience determine if and when fine-tuning becomes a beneficial step in their development process.
Who should watch
- AI Engineers
- Product Managers
- Builders evaluating LLM strategies
- Those considering fine-tuning for cost, latency, or quality improvements
- Teams struggling with inconsistent LLM output
Key takeaways
- A common graph plots model output quality against cost-effectiveness (tokens per dollar), showing a general correlation between model size/training and quality.
- Techniques like mixture-of-agents can push performance beyond the standard frontier by using more compute at inference, but are often too expensive for direct use.
- Fine-tuning allows developers to move this performance frontier outward, achieving higher quality outputs for a given budget than base models.
- The primary trade-offs for fine-tuning are the significant upfront time investment for data preparation and training, and a loss of general flexibility as the model becomes specialized.
- The decision to fine-tune should be driven by specific needs: prohibitive costs of using large base models, the need for lower latency, or a requirement for highly consistent output quality.
- Starting with a prompted model is often advisable for iteration speed and to accurately capture the input data distribution before committing to fine-tuning.
- The fine-tuning process involves four main steps: data preparation (logging prompts, human review if needed), model training (often simplified by current tools), evaluation (inner-loop for quick feedback, outer-loop for business metrics), and deployment.
- Tools and infrastructure for fine-tuning, evaluation, and deployment have matured significantly, making the process more accessible than in the past.
Notable quotes
*My guess is that most people in this audience as of right this moment probably do not need fine tuning.*
*The pitch for fine tuning is that it gives you this ability to move outside that Pareto Frontier and for whatever budget you have, you end up with a better model.*
*The trade-off that you typically run into with fine tuning is there there's a couple of different things. The biggest one honestly is just the time up front it takes.*
Unofficial community note. Prefer the recording for nuance.