World's Fair 2025

Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith

Taylor Jordan Smith

Overview

This talk addresses the critical need for robust evaluation and benchmarking strategies when deploying large language models (LLMs) into production. It highlights the inherent complexities and potential pitfalls of generative AI, emphasizing that scalability, reliability, and safety are paramount. The presentation introduces practical tools and methods to assess LLM performance, ensuring that models meet enterprise-level requirements before and during deployment.

Who should watch

• AI Engineers
• Product Managers
• Builders working with LLMs in production
• Those facing challenges in scaling and ensuring the reliability of AI systems
• Individuals seeking to understand and mitigate risks associated with LLM deployment

Key takeaways

• Deploying generative AI in production is complex, requiring careful consideration of policy restrictions, legal risks, bias, cost, performance, and knowledge cutoffs.
• Inference at scale is a major bottleneck; purpose-built inference engines like TRT-LLM, vLLM, and others are crucial for handling real-world traffic efficiently.
• Evaluating LLM performance is challenging, involving manual setup, significant compute load, ensuring dataset compatibility, resource optimization, and difficult cost estimations.
• Benchmarking is a subset of evaluation, specifically comparing models on controlled datasets and tasks, while evaluation is a broader assessment of end-to-end model performance.
• An incremental approach to LLM evaluation is recommended, starting with specific components like retrieval or latency and gradually expanding to full system evaluations.
• A tiered evaluation pyramid can guide implementation, starting with system performance and GPU utilization, moving to factual accuracy (e.g., MMLU), and then addressing safety, bias, and custom application-specific metrics.
• Tools like GuideLLM can be used for system performance benchmarks (latency, throughput), while lm-eval-harness and Promptfoo offer capabilities for factual accuracy, safety, and custom evaluations.
• Integrating evaluation frameworks into CI/CD pipelines, similar to software engineering practices, is essential for continuous improvement and monitoring of LLM performance in production.

Notable quotes

Watch on YouTube →

Unofficial community note. Prefer the recording for nuance.