The State of AI Code Quality: Hype vs Reality — Itamar Friedman, Qodo

This talk explores the evolution of evaluating AI models for coding tasks, from simple code snippets to complex codebases. It highlights challenges like data contamination and brittle test suites, proposing dynamic evaluation sets and LLM-based judges to ensure reliable and relevant assessments as AI capabilities advance. The discussion covers various stages of coding evaluation, emphasizing the need for benchmarks that reflect real-world performance and adapt to the rapid progress in AI.

This talk introduces Spec-Driven Development as a method to enhance AI agent development, focusing on improving control, code quality, and reliability. It proposes a structured workflow that compresses the software development lifecycle, moving from prompt to requirements, design, and execution, with an emphasis on creating reproducible results. The approach aims to integrate established software engineering practices with AI capabilities to tackle more complex problems.

The talk posits that the traditional Integrated Development Environment (IDE) is becoming obsolete, predicting that by 2026, most code will be generated by advanced AI systems overseen by engineers. This shift moves from current tools like code assistants, which are seen as powerful but potentially dangerous like power tools, towards more precise, automated systems akin to CNC machines. The core argument is that AI will fundamentally reshape technology organizations and the economy, enabling unprecedented productivity and new ways of building software.

This talk draws parallels between building self-driving cars and developing AI agents, emphasizing that agents, like robots, operate in complex environments and require more than just sophisticated models. The core thesis is that the 99% of work beyond the AI model—encompassing infrastructure, tooling, feedback loops, and robust offline stacks—is crucial for reliable agent development and deployment.

Poolside is developing AI models from scratch, focusing on pairing next-token prediction with reinforcement learning to bridge the gap between models and human intelligence. Their second-generation model, Malibu agent, is designed for high-consequence code environments, demonstrating capabilities in code conversion, testing, and feature implementation. The company emphasizes building foundational intelligence and scaling it, with plans for public API access and integration with platforms like Amazon Bedrock.