Max Ryabinin

This talk details Together AI's research into training large context models, specifically aiming to break the 5 million token barrier. The core challenge lies in overcoming memory and computational bottlenecks inherent in standard transformer architectures when dealing with extremely long sequences. The research explores and combines various techniques to enable efficient training at unprecedented context lengths.