World's Fair 2026
Respect The Process - Andrew Dumit, Watershed Technology Inc.
Overview
This talk addresses the challenges of deploying coding agents for complex tasks, particularly in domains like sustainability where expert judgment and nuanced processes are critical. It emphasizes that while coding agents offer significant power and flexibility, their unconstrained execution can lead to unpredictable and potentially erroneous outcomes. The core thesis is that to effectively leverage coding agents, especially when direct answer verification is difficult, it's essential to implement a "harness" that constrains the agent's *effects* rather than its reasoning, ensuring process integrity and verifiable outputs.
Who should watch
- AI Engineers working with coding agents or agentic workflows.
- Product Managers and builders evaluating the deployment of AI agents for complex tasks.
- Developers facing challenges with agent consistency, hallucination, or output verification.
- Anyone building systems where the integrity of the process is as important as the final answer.
Key takeaways
- In domains requiring expert judgment, like calculating product carbon footprints, validating only the final answer is insufficient; the process must be verifiable.
- Coding agents, while powerful for exploration and complex edits, can exhibit inconsistency, hallucinate, and produce outputs that are difficult for non-technical users to review.
- A "harness" approach is proposed, focusing on constraining the agent's effects through a typed SDK and controlled execution, rather than limiting its reasoning.
- The typed SDK acts as the sole interface for critical edits, enforcing structure and preventing conflicts, while deterministic execution by the system guarantees the integrity of the final output.
- This controlled process allows for the creation of review artifacts that are understandable even to non-coders, making the agent's work traceable and replayable.
- Continuous improvement of agent performance on complex tasks is still achievable through prompt engineering, better tool design, and task decomposition, even with process constraints in place.
- The harness ensures that even when an agent errs or the ground truth is ambiguous, the underlying process remains valid and understandable.
Notable quotes
*In these cases, there are many ways to get the right answer the wrong way and there are also many right answers that experts will disagree on.*
*And so, you have to verify the process in addition to the answer because the answer is really only justified in so far as it the process that produced that answer is correct.*
*We learned that unconstrained code is quite scary.*
Unofficial community note. Prefer the recording for nuance.