Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, OpenClaw

This talk focuses on the practical aspects of evaluating and improving AI agents, moving beyond simple "vibe checks" to implement robust testing frameworks. It emphasizes that effective evaluation is crucial for shipping reliable AI applications, especially agents, which are prone to cascading failures. The session introduces methods for capturing agent behavior through tracing, categorizing failures, and implementing various types of evaluations (code-based, LLM-as-judge, human) to ensure agents perform as expected and to drive iterative improvements.

This talk explores the integration of coding agents, specifically the OpenClaw agent, into products. It emphasizes that coding agents are becoming a fundamental building block for software systems and highlights the Pi framework as an excellent tool for experimentation and development. The presentation demonstrates how agents can be leveraged to automate complex workflows, such as processing sales proposals, by interacting with various tools and systems.

This talk introduces the concept of agentic engineering, shifting the paradigm from merely using AI tools to actively collaborating with them. It highlights the evolution of AI in software development from simple line completion to sophisticated agents capable of executing complex tasks. The core thesis is that understanding and managing AI agents as collaborators, akin to junior developers, is crucial for maximizing their potential and navigating the complexities of modern software development.

This talk addresses the challenges of AI evaluations, which are currently scattered, quickly become outdated, and often lack transparency and verifiability. The speakers propose solutions to democratize the evaluation process, enabling a broader community to contribute to and benefit from robust AI benchmarking. Their work aims to foster more equitable AI development by allowing diverse expertise to shape evaluation standards.

This talk explores agentic search as a critical component of context engineering for Large Language Models (LLMs). The core thesis is that effective context curation for LLMs relies heavily on sophisticated search mechanisms, moving beyond simple retrieval to agent-driven exploration of diverse information sources. The presentation highlights the evolution from basic RAG to agentic RAG and emphasizes the challenges and strategies for building robust search tools that agents can effectively utilize.