World's Fair 2025
From Copilot to Colleague: Trustworthy Agents for High-Stakes - Joel Hron, CTO Thomson Reuters
Joel Hron , CTO Thomson Reuters
Overview
This talk explores the evolution of AI assistants from being merely helpful to becoming productive agents capable of making judgments and decisions. It emphasizes that agency in AI is not a binary state but a spectrum that can be tuned based on use case, risk tolerance, and user expectations. The presentation highlights the challenges and lessons learned in building trustworthy AI systems for high-stakes professional environments, particularly within legal, tax, and global trade sectors.
Who should watch
- AI engineers and developers building agentic systems.
- Product Managers and leaders defining AI product strategy.
- Builders seeking to integrate AI into complex, domain-specific workflows.
- Professionals concerned with trust, accuracy, and risk in AI applications.
- Those interested in leveraging legacy systems with modern AI.
Key takeaways
- The shift in AI assistant goals is from helpfulness to productivity, requiring agents to produce output and make decisions.
- AI agency is best viewed as a spectrum of tunable dials, including autonomy, context, memory, and coordination, allowing for customization based on use case.
- Evaluating AI systems, especially agentic ones, is challenging due to inherent variability in human judgment and the difficulty in tracing agent drift.
- Legacy applications, rather than being discarded, can be decomposed into tools that agentic systems can leverage, revitalizing existing infrastructure.
- Building the entire system first, rather than focusing on minimal viable products (MVPs), can provide a clearer understanding of which components require optimization.
- Agentic systems can be used to extract data from source documents, map it to specific engines, and generate end-to-end outputs like tax returns or legal research reports.
- Trust in AI is built through determinism and expected outcomes, which is difficult to achieve with highly variable AI systems, necessitating rigorous rubrics and human evaluation.
- Unique company assets, such as domain expertise and proprietary content, are crucial for creating differentiation in AI product development.
Notable quotes
*Dont build agentic tools for law firms, build law firms of agents.*
*We're not asking assistants to just be helpful anymore. We're asking them to actually produce output.*
*Agency is not a binary thing but as a lever that you can dial up or down.*
Unofficial community note. Prefer the recording for nuance.