Code 2025
Hard Won Lessons from Building Effective AI Coding Agents – Nik Pash, Cline
Overview
This talk argues that the effectiveness of AI coding agents is primarily determined by the underlying model's capability, not by complex engineering scaffolds. Frontier models, when unhindered, can outperform many agent combinations. The core message is to simplify agent engineering and focus on improving model training through robust benchmarks and reinforcement learning environments derived from real-world coding tasks.
Who should watch
- AI Engineers
- Product Managers
- Builders working with AI agents
- Those seeking to improve AI model performance
- Individuals frustrated with overly complex agent architectures
Key takeaways
- Frontier models increasingly bypass the need for elaborate scaffolding like RAG or complex tool-calling systems, as these abstractions can hinder performance.
- Model capability, rather than agent engineering cleverness, is the main driver of performance. Benchmarks and RL environments are crucial for advancing model capabilities.
- Real-world coding tasks, when transformed into standardized RL environments, are essential for training models to handle complex problems and improve reliability.
- The process of creating RL environments from real-world data can be significantly automated, shifting the bottleneck from engineering to collecting high-quality tasks.
- Benchmarks should focus on testing the spirit and outcome of a task, not on overprescribing methods based on ground truth solutions.
- Many major AI labs collect similar data internally but do not share it, slowing down overall research progress.
- Klein is launching Klein Bench, an open-source initiative to create a benchmark based on real software development tasks, encouraging community contribution.
- Contributing to Klein Bench involves using the Klein provider on open-source projects and opting in, allowing real-world struggles to become benchmark candidates.
Notable quotes
*Capability beats scaffolding. If you get out of the models way, it will perform just fine.*
*The lesson here is relentless. A perfect example of what I'm talking about is Gemini 3.0 released this week and it immediately dominated terminal bench leaderboards with no agentic harness supporting it at all.*
*Models don't improve without this data and keeping them closed is slowing down Frontier Research.*
Up next · Coding agents in production
Watch next
The human skill that remains: wielding agents, not typing faster.
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / AmpUnofficial community note. Prefer the recording for nuance.