World's Fair 2026
Perception Agents — Antje Barth, Amazon AGI Lab
Overview
This talk introduces perception agents, a new paradigm in human-agent collaboration that moves beyond text-based interactions. Current agents often require detailed textual prompts and struggle with dynamic visual interfaces or unexpected application behaviors. Perception agents aim to bridge this gap by enabling AI to see, understand, and interact with computer interfaces similarly to humans, using clicks and keystrokes to act upon visual information.
Who should watch
- AI engineers building agentic systems
- Product Managers exploring new AI capabilities
- Builders seeking to integrate AI into dynamic software environments
- Those interested in overcoming the limitations of current text-based AI agents
- Individuals focused on improving human-AI collaboration in software development
Key takeaways
- Traditional AI agents are limited by their inability to perceive and interact with visual interfaces dynamically.
- Perception agents are designed to see computer screens, reason about visual content, and perform actions like clicks and keystrokes.
- This visual understanding allows agents to navigate changing UIs and handle unexpected application states, unlike current text-dependent models.
- The development of perception agents addresses the architectural gap between agents that perform well in demos and those that can function effectively in real-world software workflows.
- This approach enables agents to collaborate more naturally with human teams by mirroring human interaction methods with computers.
Notable quotes
*Agents most teams ship today still wait for us to type a paragraph to explain what we're looking at.*
*Perception agents close it: they see and use computers the way people do, reason about what they see, and act with clicks and keystrokes.*
Unofficial community note. Prefer the recording for nuance.