World's Fair 2026
Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo
Overview
This talk argues for designing AI systems that encourage discernment rather than mere approval. It highlights the phenomenon of cognitive surrender, where humans increasingly accept AI outputs without critical evaluation, potentially leading to errors and reduced human reasoning. The core thesis is that by engineering the human-AI interaction loop, developers can elicit more critical engagement, leading to higher quality data and more effective AI systems.
Who should watch
- AI Engineers and Developers building AI-powered tools.
- Product Managers seeking to improve user interaction with AI features.
- Builders concerned about automation bias and the reliability of AI outputs.
- Anyone working on systems where human oversight is critical but prone to error.
Key takeaways
- Cognitive surrender is a growing issue where humans accept AI output with minimal scrutiny, as seen in studies where participants accepted incorrect AI answers 80% of the time.
- In a Duolingo English Test study, human proctors accepted 50% of false AI cheating signals, demonstrating automation bias.
- Simple changes to interface copy, emphasizing the AI signal as preliminary and requiring independent evidence, significantly increased rejection rates of false alarms.
- The human-AI interaction loop is cyclical, not linear; designing this interaction is key to eliciting better human behavior and collecting valuable data.
- Structured interactions, like inline markup for writing feedback or breaking down coding agent changes, yield higher quality data than simple binary accept/reject signals.
- Friction should be intentionally built into high-stakes interactions to encourage deliberate thought, while low-stakes interactions should be frictionless for a seamless user experience.
- Every interaction with an AI system can serve as a data label; capturing nuanced feedback beyond simple yes/no is crucial for model improvement.
- Engineering the interaction, rather than solely focusing on model improvement, is often the most effective way to enhance AI system efficacy and data quality.
Notable quotes
*When a human foregoes deliberation and adopts AI output as their own with minimal scrutiny.*
*The human in the loop isn't thinking. Build AI systems for discernment, not approval.*
*Sometimes the fix is not a better model or more oversight, it's just engineering the interaction itself.*
Unofficial community note. Prefer the recording for nuance.