World's Fair 2025
What We Learned from Using LLMs in Pinterest — Mukuntha Narayanan, Han Wang, Pinterest
Mukuntha Narayanan , Han Wang , Pinterest
Overview
This talk details Pinterest's experience integrating Large Language Models (LLMs) into their search relevance system. The core thesis is that LLMs significantly improve relevance prediction, and that techniques like knowledge distillation are crucial for productionizing these models at scale. The presentation also highlights the value of synthetic captions and user engagement data as content annotations.
Who should watch
- AI Engineers
- Product Managers
- Builders working on search relevance
- Teams looking to scale LLM-based systems
- Those interested in leveraging user data for model improvement
Key takeaways
- LLMs demonstrate strong performance in relevance prediction for search queries and pins, outperforming previous in-house models.
- Incorporating LLM-generated synthetic image captions and user engagement data (like board titles and past search interactions) enhances content representation and relevance.
- Knowledge distillation is an effective method to scale powerful LLM "teacher" models into smaller, efficient "student" models for production deployment.
- The distilled student model, a bi-encoder architecture, uses both LLM components and traditional features like BM25 for efficient real-time inference and caching.
- Relevance-tuned LLMs produce rich semantic representations that are valuable for various applications beyond search, such as home feed and related pins.
- Multilingual LLMs show promise in transferring knowledge across languages, enabling relevance improvements even in regions with limited initial training data.
Notable quotes
LLMs are quite good at relevance prediction.
LLM generated captions and user actions can be quite useful content annotations.
Knowledge distillation is a very good way to scale and efficiently serve models online.
Unofficial community note. Prefer the recording for nuance.