World's Fair 2026
Design Patterns for AI Trust: Juries, Libraries, and Agent Tiers — Alex Bauer, Upside.tech
Overview
This talk introduces design patterns for building trust in AI agents, drawing parallels to human collaboration and decision-making processes. The core thesis is that managing AI agents effectively involves applying principles similar to how teams of humans are managed, particularly in complex or ambiguous situations. The presentation emphasizes that AI agents, like people, benefit from clear guidance, access to knowledge, and structured evaluation methods to ensure reliable and trustworthy outputs.
Who should watch
- Product Managers and Engineers looking to build reliable AI-powered features.
- AI Engineers seeking practical strategies for agent development and deployment.
- Builders and decision-makers concerned with AI hallucination and trust issues.
- Teams struggling with data interpretation and complex business questions.
- Anyone interested in applying human management principles to AI agents.
Key takeaways
- AI agents can empower non-technical teams by providing capabilities previously requiring extensive coding, akin to how the bicycle increased mobility.
- A significant challenge with AI is not hallucination, but the generation of plausible-sounding incorrect answers, leading to trust issues.
- A practical approach to prompting AI agents is to use commander's intent, explaining the "why" behind a task to improve outcomes, mirroring effective human management.
- Scaffolding AI agent work is crucial; define structures and provide necessary context (e.g., anchor assets, product capabilities) before tasking the agent.
- The concept of a "radiant librarian" can provide agents with just-in-time knowledge, including company-specific documentation and schema of past queries, to ensure accurate data interpretation.
- For tasks without a single empirically correct answer, a "jury and judge" workflow can be employed, where multiple independent analysts provide input, and a consensus judge weighs their reasoning.
- The quality of the AI model and its underlying harness significantly impacts results; avoid using basic interfaces for critical tasks and opt for powerful models with advanced features.
- *Friends don't let friends use really bad harnesses or low intelligent models for important work.*
Notable quotes
The speaker suggests managing AI agents like other humans: *When in doubt, manage your agents like other humans.*
A key prompting technique is commander's intent: *Use commander's intent when you prompt. This is something that comes out of the armed forces doctrine, but basically tell your agents why you want them to do something, and they will do it a lot better.*
On agent capabilities: *Find something that is at least tier two here and tier two means it's got to be a powerful model.*
Unofficial community note. Prefer the recording for nuance.