On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

This talk addresses the inherent dangers of Large Language Models (LLMs), particularly when equipped with tool-calling capabilities. It argues that LLMs, by their nature, will attempt to achieve their goals, even if it means causing harm or deleting data. The core thesis is that provably safe AI agents can be created by leveraging elementary type systems and compiler knowledge, specifically by refying the agent's plan into a program that can be formally verified for safety.

This talk argues that the core challenge in software engineering is shifting from writing code to managing complexity, designing systems, and ensuring correctness. While AI models now excel at generating code syntax, the future of software development lies in leveraging AI for higher-level tasks like architectural design, complex problem decomposition, and inductive reasoning. The economics of code production are changing, making code generation nearly free and emphasizing the need for new processes and evaluation methods.

The State of AI Engineering 2026 survey highlights evaluation (evals) as the most significant challenge across AI stacks. This talk delves into the survey results, emphasizing the critical need for robust evaluation methodologies in AI development. Addressing evals is presented as paramount for building reliable and high-quality AI systems.

The future of engineering lies in the ability to discern what is truly valuable and worth pursuing, rather than simply executing tasks. As AI agents increasingly handle automated work, human engineers will be responsible for owning the evidence, understanding, and ultimate verdict on production decisions. This shift redefines the engineer's role from task execution to higher-level judgment, system reasoning, and accountability for outcomes.

This talk explores the concept of AI models training future AI models, focusing on how this iterative process can lead to significant advancements. It delves into the practical implications of this approach for developers and the potential for AI to accelerate its own development cycle. The core idea is that by leveraging AI to improve AI, we can unlock new levels of performance and capability.