World's Fair 2026
Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua
Overview
This talk introduces advancements in AI agents for computer use, moving beyond earlier, screen-dominating models to more integrated and efficient background operations. It highlights the development of tools like Quad Driver for seamless OS interaction across platforms and Kuabench for robust agent evaluation. The discussion also touches upon optimizing infrastructure for agent training to reduce costs and improve GPU utilization.
Who should watch
- AI engineers developing or evaluating agents.
- Product Managers exploring agent capabilities for new products.
- Builders seeking to integrate AI agents into desktop applications.
- Researchers working on agent performance and reliability.
- Anyone interested in the practical application of AI for computer automation.
Key takeaways
- Quad Driver enables AI agents to interact with operating systems in the background across Mac OS, Windows, and Linux, using accessibility trees or screenshot analysis.
- Kuabench is a framework for evaluating AI agents' computer use capabilities, featuring setup, oracle, and evaluator functions, with over 130 verifiable tasks.
- Testing on Kuabench revealed that top agents struggle with complex tasks, particularly starting from a blank slate, with success rates dropping significantly.
- Using Quad Driver as the agent's computer tool can improve task pass rates and reduce token usage compared to built-in tools.
- A rigorous evaluation pipeline is employed to test and break environments before tasks are added to Kuabench, ensuring eval trustworthiness.
- Agent world models can be measured by their ability to predict the computer's state at any point during an execution trajectory.
- Optimizing RL training infrastructure with a demand-based autoscaler for sandbox pools can significantly reduce GPU idle time and costs.
- This approach allows for instant sandbox availability across multiple platforms, maximizing GPU utilization.
Notable quotes
*With cooler driver we gave an agent hands but then the question becomes how can you trust the agent to use those hands correctly.*
*The top agent that we tested only got a full pass on six out of 25 of these tasks.*
*Your GPU really isn't doing anything useful here and you know I don't know if you've heard but GPU time is pretty expensive right now.*
Unofficial community note. Prefer the recording for nuance.