Will Agent evaluation via MCP Stabilize Agent Networks? - Ari Heljakka

This talk explores what it means for Model Communication Protocol (MCP) to be enterprise-ready, moving beyond basic tool-use capabilities to address the complexities of production environments. It highlights the evolution from simple chatbot interactions to sophisticated AI agents that require robust security, management, and scalability, particularly when integrating with internal enterprise systems. The core thesis is that while MCP offers a standardized way for AI to interact with external resources, achieving enterprise readiness involves overcoming significant challenges in authentication, authorization, and operational management.

This talk focuses on mastering AI evaluation, moving from initial development to production. It emphasizes that even the best large language models (LLMs) require a robust testing framework due to issues like hallucinations and performance degradation with changes. Effective evaluation helps answer critical questions about model selection, cost-effectiveness, brand consistency, and ongoing improvement, ultimately reducing development time, costs, and enabling faster iteration.

The core thesis of this talk is that 2025 will finally be the year for robust AI evaluation, driven by the convergence of several key factors. The increasing understanding of AI by business leaders, coupled with budget shifts towards generative AI projects, has set the stage. Crucially, the rise of agentic systems, which make decisions and take actions, introduces significant complexity and risk, making rigorous evaluation a necessity rather than an option.

This talk emphasizes that robust evaluations are the most critical, yet often overlooked, component for successfully scaling generative AI workloads. The speaker argues that evaluations are not merely for measuring quality but are primarily a tool for discovering problems within AI systems. Implementing a strong evaluation framework is presented as the key differentiator between a project that remains a science experiment and one that achieves production-level success and scalability.

This talk explores the automation of business processes using Large Language Models (LLMs) through two key protocols: A2A (Agent-to-Agent) and MCP (Model Context Protocol). A2A facilitates communication between remote agents, enabling specialization and parallel processing, while MCP acts as a standardized interface for agents to access context and tools, akin to a USB-C for AI. The workshop demonstrates how to integrate these protocols to build multi-agent systems triggered by webhooks, automating tasks like bug reporting and information dissemination.