Session brief
Iterating on LLM apps at scale Learnings from Discord: Ian Webster
Overview
This talk discusses learnings from building and scaling LLM applications at Discord, focusing on the challenges of safety, security, and legal considerations. The core thesis is that robust evaluation (evals) and a strong eval culture are crucial for mitigating risks and enabling the successful deployment of LLM products at scale. The speaker emphasizes a pragmatic, developer-first approach to evals, treating them like unit tests to ensure fast, reliable feedback loops.
Who should watch
- AI Engineers
- Product Managers
- Builders working on LLM applications
- Those concerned with LLM safety, security, and legal compliance
- Teams looking to scale LLM deployments
Key takeaways
- The most significant blockers for LLM product launches are often security, legal, and policy concerns, not the models themselves.
- Quantifying and mitigating risks *before* production is essential, as failures at scale can be catastrophic.
- Treat LLM evals as simple unit tests: break them down into small, deterministic components that run quickly and locally.
- Resist the urge to over-engineer prompts; sometimes less is more, allowing the LLM room for reasonable decision-making.
- Foster an "eval culture" where running evals is a quick, reflexive part of the development process, integrated into PRs and CI/CD.
- Leverage existing observability tools (like DataDog) for LLM metrics rather than adopting entirely new, complex solutions.
- Red teaming is critical, especially for public-facing applications, to uncover application-specific jailbreaks and harmful failure modes.
- Consider both pre-deployment risk assessment and live filtering for safeguarding LLM applications.
Notable quotes
*The biggest repeat launch blockers were security, legal, safety, and sometimes policy.*
*If we want LLMs to achieve their full potential, especially in the enterprise, we need ways to measure and mitigate these risks.*
*Treat them as unit tests; figure out the specific parts of your system.*
Unofficial community note. Prefer the recording for nuance.