Evals Are Broken, Use Them Anyway — Ara Khan, Cline

This talk introduces Pydantic AI's agent optimization capabilities, focusing on the Jeppa library for prompt optimization and managed variables for dynamic configuration. The core idea is to systematically improve agent performance by iteratively refining prompts and configurations, demonstrated through a practical example of analyzing political family ties in MP data. The presentation highlights how these tools can lead to more accurate and efficient AI agents.

This talk addresses the challenges of AI evaluations, which are currently scattered, quickly become outdated, and often lack transparency and verifiability. The speakers propose solutions to democratize the evaluation process, enabling a broader community to contribute to and benefit from robust AI benchmarking. Their work aims to foster more equitable AI development by allowing diverse expertise to shape evaluation standards.

This talk explores the effectiveness of semantic code retrieval for AI coding assistants, specifically benchmarking it against traditional grep-based methods on Claude Code. The core thesis is that while grep is simpler and often sufficient, semantic search, powered by vector embeddings, can significantly improve precision and the agent's ability to locate relevant code, especially for complex tasks. This approach offers a form of cached compute, reducing redundant processing and potentially leading to better accuracy and user satisfaction.

This talk explores how to move beyond traditional code coverage metrics to ensure true functionality testing, especially in the context of AI-assisted development. It argues that while AI can increase code output, its true productivity gains are amplified by clean codebases and robust testing. The presentation introduces Playwright as a tool for end-to-end browser testing, demonstrating how it can be integrated with AI agents to accelerate development workflows, particularly within a test-driven development (TDD) framework.

This talk addresses the quality of code generated by Large Language Models (LLMs) for enterprise use. While LLMs show high functional correctness on standard benchmarks, they often produce code that is verbose, complex, insecure, and difficult to maintain. The presentation introduces an evaluation framework and a proposed agent-centric development cycle to address these shortcomings and ensure LLM-generated code meets engineering standards.