Session brief
LLM codegen fails and how to stop 'em — Danilo Campos, PostHog
Overview
This talk addresses common failure modes in Large Language Model (LLM) code generation and offers strategies to mitigate them. The core thesis is that while LLMs can automate complex tasks like software integration, their outputs are prone to issues like outdated knowledge, architectural inconsistencies, and unpredictable behavior. By understanding these failure points and implementing specific techniques, developers can improve the reliability and effectiveness of autonomous coding agents.
Who should watch
- AI engineers and developers working with LLM-powered coding tools.
- Product managers and builders evaluating the use of agents for software development tasks.
- Anyone experiencing unexpected or incorrect outputs from code generation models.
- Teams looking to improve the reliability and consistency of agent-driven development processes.
Key takeaways
- **Model Rot:** LLMs are trained on data snapshots and can become outdated. To combat this, use Retrieval Augmented Generation (RAG) or inject fresh documentation directly into the agent's context.
- **Architectural Drift:** Agents may generate code with suboptimal or inconsistent architecture. Maintain "model airplanes" – simplified, correctly shaped examples of integrations – to guide agents toward desired patterns.
- **Unpredictable Paths:** Agents can find numerous ways to solve a problem, leading to support burdens. Limit improvisation by " এর breadcrumbing" the agent, guiding it through a sequence of smaller, well-defined steps rather than providing a single, large instruction.
- **Human Error:** Developers themselves can introduce errors through contradictory instructions or incorrect context. Implement inference-time interrogation of agent runs to identify and correct these issues.
- **Security Shenanigans:** Agents running on user machines pose security risks. Implement fine-grained control over tool usage and data access, such as creating specific tools for sensitive operations like reading environment variables.
- **Value Shift to Prose:** Traditional coding is becoming a depreciating asset. The real value now lies in well-crafted prose and instructions that LLMs can interpret and act upon, even as models improve.
- **Agent as an Octopus:** Avoid over-constraining agents. Instead, focus on providing sufficient, well-sequenced information to enable them to solve problems effectively.
- *Plain text prose is where so much of our value now lives.*
Notable quotes
Danilo Campos: *The wizard that makes everybody so happy is 90% markdown files, 8% tools for delivering and processing markdown files, And then the rest is like Agent Harness stuff.*
Danilo Campos: *An agent is an octopus, right? It can wiggle, it can squeeze into tight corners, it can maneuver itself around problems. You do not want to over-constrain the agent in its ability to get problems done.*
Unofficial community note. Prefer the recording for nuance.