World's Fair 2024
Lessons From A Year Building With LLMs
Overview
This talk, delivered by a collective of six AI engineers, distills a year of practical experience building with Large Language Models (LLMs). The core thesis is that successful LLM application development hinges not on proprietary models, but on strategic product building, robust operational processes, and meticulous tactical execution. The speakers emphasize a continuous improvement loop, drawing parallels to established software engineering and machine learning practices, to navigate the complexities and uncertainties inherent in LLM development.
Who should watch
- AI Engineers
- Product Managers
- Builders and developers working with LLMs
- Those seeking to move beyond LLM demos to production-ready products
- Teams struggling with LLM application quality and iteration
Key takeaways
- The LLM model itself is rarely the product moat; focus on leveraging product expertise and building in your niche.
- Treat LLMs as interchangeable SaaS products, ready to switch to superior alternatives as they emerge.
- An excellent LLM-powered application is fundamentally an excellent product that solves a job to be done and enhances the user.
- Continuous improvement, driven by evaluation and data, is crucial, mirroring principles from MLOps, DevOps, and the Lean Startup movement.
- *Value is only created when metal gets bent.* Focus on delivering tangible value to users rather than getting lost in tool construction.
- Evals are objectives, not just metrics, and should be central to the iterative improvement loop.
- Shipping early and iterating based on real user interactions and feedback is vital for finding product-market fit.
- Plan for the future by projecting technological advancements, such as decreasing LLM inference costs, to enable currently uneconomical applications.
Notable quotes
*The model is not your moat for almost no one in this audience.*
*An excellent LLM powered application is an excellent product.*
*Value is only created when metal gets bent.*
Unofficial community note. Prefer the recording for nuance.