World's Fair 2025
How to Improve your Vibe Coding — Ian Butler
Overview
This talk addresses the current limitations of AI coding agents in identifying and fixing bugs, highlighting their tendency to generate false positives and struggle with complex codebases. It proposes practical strategies for developers to improve the effectiveness of these agents, focusing on structured rules, better context management, and leveraging advanced "thinking" models. The core thesis is that while agents show promise, their current implementation requires careful guidance and configuration to be truly useful for developers.
Who should watch
- AI engineers and developers using AI coding assistants.
- Product Managers evaluating the reliability of AI-powered development tools.
- Builders seeking to improve their workflow and reduce bugs in their code.
- Anyone experiencing alert fatigue from AI-generated suggestions.
Key takeaways
- Many popular AI coding agents exhibit low true positive rates for bug detection, with some having over 97% false positives, leading to alert fatigue and reduced trust.
- To improve agent performance, developers should implement bug-focused rules, providing scoped instructions on specific security and logical issues, referencing resources like the OWASP Top 10.
- Explicitly naming bug types (e.g., SQL injection, off bypasses) in agent rules primes them to look for specific vulnerabilities more effectively.
- Context management is crucial; agents can lose logical links in large codebases. Developers should manage context windows carefully, provide diffs, and ask agents to inventory code components.
- Thinking models demonstrate superior bug-finding capabilities compared to non-thinking models, often exhibiting deeper reasoning chains.
- Agents lack holistic file understanding, leading to variability in bug detection across runs; users may need to run agents multiple times for comprehensive results.
- Requiring fix validation, such as ensuring generated tests pass, is essential to confirm that agents have actually resolved identified bugs.
- Structured rules, rather than vague requests, significantly improve agent output quality and mitigate alert fatigue.
Notable quotes
*Most popular agents are terrible at finding bugs.*
*Thinking models rock. They are just significantly better at this problem.*
*Structured rules eliminate the vague check for bugs requests and produce alert fatigue.*
Unofficial community note. Prefer the recording for nuance.