Session brief
Patrick Dougherty: How to Build AI Agents that Actually Work
Overview
This talk focuses on practical lessons learned from rebuilding a product around AI agents, emphasizing the importance of enabling agents to reason rather than solely relying on the underlying model's knowledge. Key to this approach is the concept of the Agent Computer Interface (ACI), which involves carefully designing the structure and format of tool calls and their responses. The speaker argues that focusing on ACI iteration and the surrounding ecosystem, including user experience and security, is more valuable than fine-tuning models or over-reliance on abstraction frameworks for production environments.
Who should watch
- AI Engineers building agent-based systems
- Product Managers evaluating agent capabilities
- Developers seeking to improve agent reliability and performance
- Those struggling with agent reasoning and tool usage
- Teams considering multi-agent system design
Key takeaways
- An AI agent must be able to take directions, call at least one tool with a response, and autonomously reason how and when to use tools to achieve an objective.
- Focus on enabling agent reasoning over the model's inherent knowledge; use discrete tool calls for retrieval rather than inserting all context into the system prompt.
- The Agent Computer Interface (ACI), the syntax and structure of tool calls and responses, significantly impacts agent accuracy. Small tweaks here can yield substantial improvements.
- Response formatting is crucial; switching from markdown to JSON for GPT-4o and XML for Claude improved agent performance by better aligning with model expectations.
- Fine-tuning models is generally a waste of time for agent development, as it can decrease reasoning ability by overfitting to specific tasks.
- For production, prioritize building the ecosystem around the agent, including user experience and security protocols, over abstracting with frameworks like LangGraph or CrewAI, which can present blockers.
- In multi-agent systems, implement a manager agent to delegate subtasks to worker agents, limiting the total number of agents (ideally 5-8) to avoid infinite loops and ensure task completion.
- Incentivize the manager agent to achieve the overall objective, rather than forcing worker agents through predefined steps.
Notable quotes
*The agent needs to be able to take directions, have access to call at least one tool and get a response back, and should be able to autonomously reason how and when to use its tools to accomplish that objective.*
*Small tweaks to the agent computer interface can have a massive impact on the accuracy and performance of your agent.*
*The most valuable thing you can do is the set up the ecosystem around your agent including the user experience of how your user interacts with your agent and then also the connections and the security protocols that your agent has to follow in doing its work.*
Unofficial community note. Prefer the recording for nuance.