Code 2025
Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor
Overview
This talk explores the evolution of evaluating AI models for coding tasks, from simple code snippets to complex codebases. It highlights challenges like data contamination and brittle test suites, proposing dynamic evaluation sets and LLM-based judges to ensure reliable and relevant assessments as AI capabilities advance. The discussion covers various stages of coding evaluation, emphasizing the need for benchmarks that reflect real-world performance and adapt to the rapid progress in AI.
Who should watch
- AI engineers working on code generation models.
- Product Managers evaluating the performance of coding AI tools.
- Builders seeking to understand the challenges in benchmarking AI code capabilities.
- Researchers interested in the progression of AI in software development.
- Anyone concerned with data contamination and test suite robustness in AI evaluations.
Key takeaways
- Evaluating coding AI has progressed from single-line snippets to entire codebases, reflecting rapid advancements in the field.
- Data contamination and insufficient or brittle test suites are significant challenges in current coding evaluations.
- Dynamic evaluation sets, which are periodically updated, can help combat contamination and recalibrate difficulty distributions over time.
- Benchmarks should aim for construct validity, ensuring measurements accurately reflect real-world performance, especially in tasks like code optimization.
- Reward hacking, where models exploit evaluation infrastructure or overfit test distributions, is a growing concern that requires robust detection mechanisms.
- LLM judges can be leveraged to detect non-idiomatic coding patterns and other adversarial behaviors that traditional tests might miss.
- For long-horizon tasks like codebase translation, intermediate correctness signals are crucial for measuring incremental progress.
- Human-centric experimental design, considering factors like latency in real-world applications, is vital for meaningful evaluations.
Notable quotes
*The first challenge in evaluating language models these days is like data contamination.*
*As we go to more and more real world tasks, this is going to get more challenging and we need to figure ways to combat these kind of reward hacking patterns.*
*Latency is a big concern for acceptance rates.*
Unofficial community note. Prefer the recording for nuance.