Europe 2026
Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI
Overview
This talk details Together AI's research into training large context models, specifically aiming to break the 5 million token barrier. The core challenge lies in overcoming memory and computational bottlenecks inherent in standard transformer architectures when dealing with extremely long sequences. The research explores and combines various techniques to enable efficient training at unprecedented context lengths.
Who should watch
- AI engineers and researchers working on or interested in large context models.
- Developers building agentic applications or systems requiring long-term memory and temporal consistency.
- Builders and PMs exploring the capabilities and limitations of current LLM training.
- Anyone interested in optimizing transformer training for memory and computation.
Key takeaways
- Training long context models is crucial for applications like AI agents and video generation, which require models to effectively process and retain information over extended sequences.
- Standard transformer architectures face two primary bottlenecks with increasing context length: quadratic computation complexity and linearly growing memory usage for activations.
- Several techniques can mitigate these issues, including DeepSpeed Ulysses for context parallelism, activation checkpointing to reduce memory footprint, and offloading activations to CPU memory.
- Arctic sequence length training allows tiling computations across the sequence length to avoid large buffer allocations.
- Together AI developed a novel optimization, dubbed U-Pipe, which further refines context parallelism by chunking attention head computations and reusing allocated buffers across iterations, enabling training up to 5 million tokens.
- This combined approach allows for training models with significantly larger context windows while maintaining competitive throughput, even at smaller scales.
- Profiling tools and detailed analysis are essential for identifying and addressing unexpected bottlenecks in long context training.
Notable quotes
*The primary reasons for that are twofold, I would say. First of all, with the explosion in popularity of agents, you can see a lot of different applications where you might want to put as many tokens as you want in your context.*
*The problem here is that if you are taking a standard transformer-based language model and trying to extend its context, you can run into two bottlenecks. Bottleneck number one is that you are faced with quadratic computation.*
*The second problem is more insidious, one might say. As you continue scaling your context, your memory keeps growing linearly, which is not as bad, but still pretty difficult to deal with, unless you apply a range of specific techniques.*
Unofficial community note. Prefer the recording for nuance.