World's Fair 2025
Building AI Products That Actually Work — Ben Hylak (Raindrop), Sid Bendre (Oleve)
Overview
This talk focuses on the practical aspects of building AI products that are reliable and effective, moving beyond theoretical discussions of evaluations. It emphasizes that while AI technology is advancing, the core challenge lies in effective communication and managing the inherent complexity and undefined behaviors of AI systems. The speakers advocate for an iterative approach to product development, where continuous refinement based on real-world user data and signals is crucial for success.
Who should watch
- AI Engineers
- Product Managers
- Builders of AI applications
- Those struggling with AI product iteration and reliability
- Teams looking to scale AI products effectively
Key takeaways
- Iteration and continuous refinement are critical for building successful AI products, rather than relying solely on initial evaluations.
- AI products still face challenges with unpredictable behavior and edge cases, especially when integrating with other tools.
- Traditional evaluation methods are insufficient; instead, focus on collecting and analyzing real-world user signals to understand product performance.
- Signals can be explicit (e.g., user feedback like thumbs up/down) or implicit (e.g., detecting refusals or user frustration).
- The Trellis framework offers a systematic approach to refining AI experiences by discretizing outputs, prioritizing based on impact, and recursively refining within focused buckets.
- Prioritizing workflows should consider not just volume but also negative sentiment and estimated achievable delta for improvements.
- Building structured, self-attributable, and deterministic workflows allows teams to iterate more quickly and reliably.
- The goal is to engineer repeatable, testable, and attributable AI magic, rather than relying on accidental success.
Notable quotes
*Iteration is actually one of the most important parts of building AI products that actually work.*
*We need some way to guide the chaos instead of eliminating it.*
Unofficial community note. Prefer the recording for nuance.