World's Fair 2025

Shipping Products When You Don't Know What they Can Do — Ben Stein, Teammates

Overview

This talk addresses the challenges of building and shipping products in the rapidly evolving AI agent space, where the capabilities of the underlying LLMs are not fully understood. It proposes a shift in product development from defining specific requirements to focusing on affordances and emergent behaviors, emphasizing the need for new tools and practices to navigate this uncertainty. The core thesis is that product management must transform to embrace discovery and adaptation in a probabilistic world.

Who should watch

Product Managers and Product Engineers grappling with uncertainty in AI product development.
Builders and engineers working with AI agents and LLM-based applications.
Anyone interested in the future of product development methodologies in the age of AI.
Teams struggling to define clear requirements for probabilistic AI systems.

Key takeaways

The inherent unpredictability of LLMs means product teams cannot fully know what their products can do, necessitating a new approach to development.
Product development should shift from specifying exact features to defining affordances, trusting that the AI will discover functionality within those boundaries.
Emergent behavior is a key aspect of AI products; teams should focus on discovering and leveraging unexpected capabilities rather than trying to pre-define all outcomes.
Evals serve as a crucial testing framework for probabilistic AI and can function as a new form of product specification, guiding development and quality assessment.
Vibe coding, or rapid prototyping and experiencing the AI interaction, is essential for understanding the feel and usability of AI products, as traditional documentation methods are insufficient.
Debugging and bug reporting in AI products are complex, requiring a new understanding of what constitutes a bug versus emergent or acceptable probabilistic behavior.
Customer interactions require a new narrative, focusing on co-creation and shared discovery of the future, as traditional visionary or honest broker roles are difficult to maintain.
The product management discipline is undergoing a profound transformation, requiring a willingness to forget past assumptions and adapt to new tools and techniques.

Notable quotes

*The product management discipline is going to undergo a transformation, a shift, an evolution, whatever you call it, that is super profound.*

*We're going to trust the LLMs. We're going to trust the agentic workflow, the work planning, like all of the things inside of our um you know, beautiful 12 factor agent.*