World's Fair 2025
Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai
Overview
This talk explores multiple frontiers in AI beyond just raw intelligence, emphasizing the trade-offs involved in accessing advanced AI capabilities. It highlights that the most intelligent models are not always the most suitable due to implications for cost, latency, and verbosity. The presentation uses benchmarking data to illustrate these trade-offs across reasoning capabilities, open-weight models, cost-effectiveness, and speed.
Who should watch
- AI engineers and builders evaluating model choices for applications.
- Product Managers and designers considering the feasibility and cost of AI features.
- Anyone interested in the practical implications of AI model performance beyond benchmarks.
- Those working with agentic systems where latency and cost are critical factors.
- Developers exploring the landscape of open-source AI models.
Key takeaways
- Reasoning models offer greater intelligence but use significantly more output tokens, leading to increased latency and cost compared to non-reasoning models.
- The gap between open-weight and proprietary model intelligence has significantly narrowed, with recent releases from Chinese AI labs like Deepseek and Alibaba showing performance close to leading proprietary models.
- The cost of accessing GPT-4 level intelligence has decreased by over 100 times since mid-2023, with some models being over 500 times cheaper to run than older frontier models.
- Output speed (tokens per second) has dramatically increased, allowing for faster responses even for highly intelligent models, which is crucial for responsive applications and agentic workflows.
- Mixture of Experts (MoE) models, inference software optimizations, and hardware improvements are driving increased efficiency and speed.
- Despite efficiency gains, the demand for compute is expected to continue rising due to larger models, the insatiable demand for more intelligence, and the increasing use of agents.
- When building applications, consider the cost structure and potential future feasibility, as costs are rapidly declining.
- It is important to measure and understand model verbosity and reasoning token usage, not just per-token pricing, to accurately assess costs.
Notable quotes
*There's more than one frontier in AI. There's trade-offs to accessing this intelligence. You shouldn't always use the leading most intelligent model.*
*The gap between open weights intelligence and proprietary model intelligence is less than it's ever been.*
*Net net, we're going to continue to see compute demand increase.*
Unofficial community note. Prefer the recording for nuance.