Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs
This talk introduces Vending-Bench, a novel evaluation framework designed to assess long-horizon agentic behavior in simulated business environments. It addresses the challenge of models acting differently when they suspect they are being tested, proposing methods to maintain reproducibility and observe emergent behaviors like collusion and power-seeking over extended periods. The work also explores the transition of AI-run businesses from simulation to the real world.