Europe 2026
Building Generative Image & Video models at Scale - Sander Dieleman, Google DeepMind
Overview
This talk provides a behind-the-scenes look at training generative image and video diffusion models at scale. It covers essential aspects from data curation and representation to modeling, architecture, training, sampling, and control mechanisms. The core thesis emphasizes that while diffusion models are powerful, their effectiveness relies heavily on careful data handling, efficient latent representations, and sophisticated sampling techniques like guidance.
Who should watch
- AI Engineers working on generative media.
- Researchers exploring diffusion model architectures and training.
- Product Managers and builders interested in the practicalities of scaling AI models.
- Anyone curious about the underlying processes of generating high-quality images and videos.
Key takeaways
- Data curation is crucial for high-quality generative model results, often being a more impactful investment than model tweaking.
- Compressed latent representations, learned via autoencoders, are essential for handling the large memory requirements of high-resolution image and video data.
- Diffusion models work by learning to denoise data through a gradual process, effectively operating in a spectral domain from coarse to fine features.
- U-Nets and Transformers are common architectures for diffusion model denoisers, with transformers leveraging knowledge from large language models.
- Guidance is a powerful technique that allows trading off sample diversity for quality, significantly improving model performance with prompts.
- Distillation in diffusion models focuses on reducing sampling steps rather than model size, enabling faster generation.
- Controlling generative models increasingly involves signals beyond text prompts, such as reference images or specific parameters like camera motion.
- Training at scale requires sophisticated parallelism and sharding techniques, with JAX offering robust tooling for this.
Notable quotes
*Time spent on improving the data is sometimes a better investment of that time than actually trying to tweak the model.*
*Diffusion is basically spectral auto-regression.*
*Guidance is a no-brainer these days. So everyone just always leaves it on.*
Unofficial community note. Prefer the recording for nuance.