Europe 2026
Fighting AI with AI — Lawrence Jones, Incident
Overview
This talk explores how AI engineers can leverage AI itself to manage the complexity of AI products. It details strategies for using internal tools to understand, debug, and improve AI systems, particularly when dealing with intricate prompt chains and complex decision-making processes. The core thesis is that effective AI development requires applying AI-powered solutions to the engineering and debugging workflows.
Who should watch
- AI Engineers
- Product Managers building AI features
- Developers working with complex AI systems
- Anyone struggling to debug or maintain AI-driven products
- Those interested in improving AI development workflows
Key takeaways
- Evals can be thought of as AI unit tests, defined in YAML, to verify prompt performance against specific criteria.
- Managing large and complex eval datasets can be challenging, especially for coding agents that have context window limitations.
- A dedicated CLI tool, eval tool, was developed to enable coding agents to effectively interact with and manage eval suites.
- Translating complex AI system UIs into file systems allows agents like Claude Code to analyze and debug interactions more effectively.
- Downloaded file system representations of AI interactions, including traces and prompts, provide rich context for agents to identify issues.
- Repeatable analysis pipelines can be built using AI agents, processing large batches of data (like daily investigations) to identify patterns and root causes of failures.
- Leveraging sub-agents for parallel analysis, storing incremental analysis in files, and combining this with codebase access enables agents to not only identify but also suggest fixes for AI system problems.
- AI should be used as effectively in internal debugging and development tools as it is in the end products themselves, prioritizing tools that work well with coding agents.
Notable quotes
*Evals are AI unit tests.*
*File systems are exceptionally good agent context.*
*You really need to be using AI just as effectively in your internal tools to try and understand these systems.*
Unofficial community note. Prefer the recording for nuance.