Ibragim Badertdinov

This talk details the practical lessons learned from evaluating coding agents on real-world software engineering tasks using the rebench leaderboard. It emphasizes the critical need for robust evaluation beyond gut feelings or limited testing, especially as models are deployed to production. The presentation highlights the challenges and methodologies involved in creating and maintaining a "fresh" and "real-world" benchmark, focusing on the complexities of software engineering tasks that require understanding repository structures, writing and running tests, and handling multi-turn interactions and tool use.