WTF Is the Context Layer? The Missing Infrastructure for Production Agents — Prukalpa Sankar

This talk explores the design principles behind creating secure and scalable agent sandbox clouds. It begins by explaining why AI models need the ability to execute code or use tools to handle tasks with verifiable rewards, such as math and coding problems. The presentation then delves into the evolution of sandbox technologies, from basic process isolation to advanced virtualization, emphasizing the critical need for robust security to protect against malicious or overzealous code execution in both research and product environments.

This talk argues that the core challenge in software engineering is shifting from writing code to managing complexity, designing systems, and ensuring correctness. While AI models now excel at generating code syntax, the future of software development lies in leveraging AI for higher-level tasks like architectural design, complex problem decomposition, and inductive reasoning. The economics of code production are changing, making code generation nearly free and emphasizing the need for new processes and evaluation methods.

This talk addresses the critical gap between raw AI agent output and a user-friendly experience, arguing that a dedicated rendering layer is essential for LLM pipelines. It emphasizes that simply displaying raw LLM responses fails to meet user expectations for intuitive and effective interaction. The core thesis is that building a robust rendering layer is key to successfully shipping AI-powered products.

This talk details OpenGov's journey in building and scaling OG Assist, an AI agent integrated across their government software products. The core thesis revolves around the strategic decision to bet on the Effect TypeScript library for building a robust agent loop, enabling significant advancements in development and production capabilities. The presentation highlights practical applications, architectural choices, and the iterative process of improving AI agent performance in a real-world enterprise environment.

This talk outlines a repeatable framework for designing and building AI systems from initial idea to production. It emphasizes that in the current AI landscape, defining product requirements, system design, and evaluation criteria are the critical, challenging aspects, rather than just coding. The framework consists of four phases: product requirements, system design, evaluation and monitoring, and optimization.