Developer Experience in the Age of AI Coding Agents – Max Kanat-Alexander, Capital One

This talk argues that AI copilots for tech architecture represent the highest ROI use case currently underutilized by organizations. While coding copilots have become standard, an architecture co-pilot can prevent costly mistakes by ensuring development efforts are directed correctly from the outset. This approach aims to move beyond tribal knowledge and gut instinct, providing data-backed guidance for architectural decisions.

Cursor Composer is an AI model designed for real-world software engineering, aiming to balance speed and intelligence. It performs better than top open-source models and rivals frontier models in intelligence, while being significantly more efficient in token generation. The development focused on creating a model that could serve as a daily driver for coding tasks, integrating features like parallel tool calling and semantic search to enhance its capabilities.

This talk explores the evolution of evaluating AI models for coding tasks, from simple code snippets to complex codebases. It highlights challenges like data contamination and brittle test suites, proposing dynamic evaluation sets and LLM-based judges to ensure reliable and relevant assessments as AI capabilities advance. The discussion covers various stages of coding evaluation, emphasizing the need for benchmarks that reflect real-world performance and adapt to the rapid progress in AI.

This talk introduces Command Code, a coding agent designed to learn and adapt to an individual developer's unique coding style and preferences, referred to as "taste." Unlike standard AI coding assistants that provide generic code, Command Code aims to internalize a developer's decision-making process, preferences for tools, and architectural choices. This is achieved through a meta-neuro-symbolic reinforcement learning approach, creating a more personalized and efficient coding experience.

This talk explores the architecture and development of coding agents, emphasizing the crucial role of the "harness" – the interface layer that connects models to users and tools. It highlights the rapid evolution of AI models and the challenges in adapting agents, proposing that building models and harnesses together, as exemplified by OpenAI's Codeex, offers a more robust and future-proof approach. The discussion also touches upon emerging patterns for integrating these agents into various products and workflows.