World's Fair 2025
POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments - Randall Hunt, Caylent
Overview
This talk shares hard-earned lessons from over 200 enterprise Generative AI deployments, emphasizing that AI is not a universal solution. It highlights the importance of understanding customer needs, robust evaluation, and efficient architecture over simply adopting the latest AI trends. The core thesis is that successful AI implementation requires a deep understanding of inputs, outputs, and user behavior, rather than relying solely on advanced models or prompt engineering.
Who should watch
- AI Engineers
- Product Managers
- Builders evaluating GenAI solutions
- Teams struggling with enterprise AI deployment
- Those seeking practical advice beyond hype
Key takeaways
- Generative AI is not a magical solution; its effectiveness depends on specific use cases and careful implementation.
- Understanding customer access patterns and how products will be used is more critical than just running evaluations or using embeddings.
- Speed and cost are crucial; slower or more expensive solutions, even with a good UI, are unlikely to be adopted.
- Prompt engineering has become highly effective, often proving more impactful than fine-tuning models, especially with recent model advancements.
- Context management is key to differentiating applications, requiring the injection of relevant user information and the optimization of context to reduce model reasoning load.
- *Embeddings alone do not a great query system make.* Faceted search and filters are necessary additions.
- *A tiny little bit of annotation can do wonders for video understanding models.* Augmenting video data, even with simple annotations, significantly improves model performance.
- Knowing your end customer and their specific environment (e.g., low connectivity, noisy surroundings) is vital for designing effective AI solutions.
Notable quotes
Randall Hunt: *Generative AI is not the the magical pill that solves everything that a lot of people seem to think it is.*
Randall Hunt: *If I were to channel my inner balmer, what I would say is eval.*
Randall Hunt: *Prompt engineering has proven unreasonably effective for us, like far more effective than I would have predicted.*
Unofficial community note. Prefer the recording for nuance.