World's Fair 2025
From Mixture of Experts to Mixture of Agents with Super Fast Inference - Daniel Kim & Daria Soboleva
Overview
This talk explores scaling large language models (LLMs) beyond traditional Mixture of Experts (MoE) architectures by introducing the concept of Mixture of Agents (MoA). MoA leverages multiple specialized LLMs, akin to agents, to collectively solve complex problems, aiming for higher intelligence and efficiency compared to monolithic models. The presentation also highlights Cerebras' hardware, designed for extremely fast inference, which enables practical implementation of MoA systems.
Who should watch
- AI Engineers
- Product Managers
- Builders experimenting with LLM architectures
- Those interested in efficient LLM inference
- Developers looking to build complex AI applications
Key takeaways
- Mixture of Experts (MoE) is an industry standard for scaling LLMs by specializing feed-forward networks into experts, allowing for larger models without proportional inference time increases.
- Mixture of Agents (MoA) extends the MoE concept by using distinct LLMs as agents, each with custom prompts, to tackle problems requiring multi-step reasoning or diverse specializations.
- MoA systems can outperform monolithic frontier models on complex tasks by aggregating the collective intelligence of multiple, potentially less powerful, individual agents.
- Cerebras hardware offers significantly faster inference speeds compared to traditional GPUs due to its architecture with a high core count and distributed on-chip memory, reducing bottlenecks.
- A hands-on workshop demonstrated building and configuring MoA systems using Cerebras' platform, including a challenge to generate optimized code.
- The MoA approach can enable non-frontier or open-source models to achieve performance exceeding that of larger, single models through intelligent orchestration.
- Challenges in MoA implementation include prompt engineering and system configuration to optimize agent collaboration for specific tasks.
- Cerebras is expanding its global data center presence and working on supporting new model architectures and multimodal capabilities.
Notable quotes
*Instead of having one monolithic fit forward network, we will create separate fit forward networks and we will call them experts.*
*A mixture of agents is basically the ability to take advantage of these earthshattering speeds from our hardware and apply them into harder problems.*
*The whole point though is that it can be better. It's just that you have to like actually like engineer it to be better.*
Unofficial community note. Prefer the recording for nuance.