Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar

This talk introduces Pydantic AI's agent optimization capabilities, focusing on the Jeppa library for prompt optimization and managed variables for dynamic configuration. The core idea is to systematically improve agent performance by iteratively refining prompts and configurations, demonstrated through a practical example of analyzing political family ties in MP data. The presentation highlights how these tools can lead to more accurate and efficient AI agents.

This talk addresses the challenges of AI evaluations, which are currently scattered, quickly become outdated, and often lack transparency and verifiability. The speakers propose solutions to democratize the evaluation process, enabling a broader community to contribute to and benefit from robust AI benchmarking. Their work aims to foster more equitable AI development by allowing diverse expertise to shape evaluation standards.

This talk explores the effectiveness of semantic code retrieval for AI coding assistants, specifically benchmarking it against traditional grep-based methods on Claude Code. The core thesis is that while grep is simpler and often sufficient, semantic search, powered by vector embeddings, can significantly improve precision and the agent's ability to locate relevant code, especially for complex tasks. This approach offers a form of cached compute, reducing redundant processing and potentially leading to better accuracy and user satisfaction.

This talk explores how to move beyond traditional code coverage metrics to ensure true functionality testing, especially in the context of AI-assisted development. It argues that while AI can increase code output, its true productivity gains are amplified by clean codebases and robust testing. The presentation introduces Playwright as a tool for end-to-end browser testing, demonstrating how it can be integrated with AI agents to accelerate development workflows, particularly within a test-driven development (TDD) framework.

This talk critiques the current state of AI model evaluations, arguing that while flawed, they remain essential tools for development. The speaker proposes a three-stage approach to using evals effectively: leveraging external evals, using them to improve internal agents, and building custom evals for specific use cases. The core message is to use evals pragmatically, understanding their limitations while still employing them for iterative improvement.