Session brief
Finetuning: 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt
Overview
This talk details how Method scaled its AI agent operations to over 500 million agents with a small engineering team. Initially, they faced challenges with manual data aggregation processes and later with the high costs and limitations of using large language models like GPT-4 for parsing unstructured financial data. The solution involved fine-tuning smaller, more efficient models to meet specific production requirements for accuracy, latency, and cost.
Who should watch
- AI engineers and builders looking to scale agentic workflows.
- Product Managers and engineers dealing with high API costs and latency issues.
- Teams struggling to extract and process unstructured data reliably.
- Organizations considering fine-tuning for cost and performance optimization.
Key takeaways
- Manual processes for data aggregation in finance are slow, expensive, and prone to human error.
- Large language models like GPT-4 are effective at parsing unstructured data but can incur significant costs and prompt engineering challenges at scale.
- Method initially incurred $70,000 in monthly costs for GPT-4 usage before optimizing.
- Scaling agentic workflows requires addressing quality, cost, and latency simultaneously.
- Fine-tuning smaller, custom models can significantly improve price-performance ratios compared to using off-the-shelf large models.
- An 8 billion parameter model, when fine-tuned, can achieve performance metrics suitable for high-volume production environments.
- Productionizing AI agents requires patience, as iterative improvements are necessary to reach desired reliability.
- Leveraging existing data from initial LLM experiments can streamline the fine-tuning process.
Notable quotes
*The status quo that we're dealing with here is a very inefficient manual process.*
*Advanced LLMs especially post GPT-4 are really good with parsing unstructured data.*
*Fine-tuning is a power tool; it does take more time and engineering investment than just prompting a model.*
Unofficial community note. Prefer the recording for nuance.