World's Fair 2024
Judging LLMs: Alex Volkov
Overview
This talk uses a courtroom drama format to highlight common pitfalls in AI engineering, particularly concerning Large Language Model (LLM) development and deployment. The core thesis is that rigorous evaluation, logging, and prompt iteration are crucial for successful LLM projects, and neglecting these can lead to significant problems. The presentation emphasizes the importance of a human-in-the-loop approach for effective LLM judging and evaluation.
Who should watch
- AI Engineers
- Product Managers
- Builders working with LLMs
- Anyone deploying LLM-powered applications
- Teams struggling with LLM evaluation and debugging
Key takeaways
- Deploying LLM applications to production without proper tracing and logging can lead to an inability to debug issues, as seen in the case of Daniel R.
- Premature fine-tuning without prompt iteration is a common mistake that wastes resources and may not yield optimal results, as illustrated by Sasha S.
- Staying updated with the rapid advancements in LLM models and tools is essential for AI engineers.
- Programmatic evaluations, while useful for simple assertions and code, are insufficient for complex conversational AI and require human oversight.
- Human-in-the-loop evaluation is critical for assessing LLM performance, especially for nuanced tasks and multi-turn conversations.
- A robust evaluation pipeline should incorporate multiple methods, including programmatic checks and human judgment, and utilize tools for visualization and experimentation.
- Developing custom evaluation criteria tailored to specific business needs is more effective than relying solely on off-the-shelf solutions.
- Tools like Weights & Biases (W&B) can simplify tracing, logging, and evaluation, enabling faster iteration and experimentation.
Notable quotes
*If you build um non-production stuff in hackathons that's fine but if you put anything of value in production you have to trace and log everything.*
*Folks, it's very important to remember that you have to iterate on prompts before you find tune.*
*The very simpleton lolms of your year are not yet escapable and so don't expect Perfection.*
Unofficial community note. Prefer the recording for nuance.