Naman Jain

This talk explores the evolution of evaluating AI models for coding tasks, from simple code snippets to complex codebases. It highlights challenges like data contamination and brittle test suites, proposing dynamic evaluation sets and LLM-based judges to ensure reliable and relevant assessments as AI capabilities advance. The discussion covers various stages of coding evaluation, emphasizing the need for benchmarks that reflect real-world performance and adapt to the rapid progress in AI.