Europe 2026
Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs
Overview
This talk provides a hands-on guide to training a transformer-based Large Language Model (LLM) from scratch using PyTorch. It covers the fundamental building blocks of LLMs, including tokenization, model architecture, and the training loop, demonstrating how to implement these components with minimal code. The session emphasizes practical application, enabling participants to train a small model locally or on cloud platforms like Google Colab.
Who should watch
- AI engineers and researchers interested in understanding LLM internals.
- Builders looking to train custom language models without relying on pre-trained weights.
- Developers seeking practical experience with transformer architectures and training pipelines.
- Individuals curious about the process of creating LLMs from the ground up.
Key takeaways
- Training an LLM from scratch involves four core components: a tokenizer, model architecture, a training loop, and inference.
- Character-level tokenization is used for simplicity in this workshop, though Byte Pair Encoding (BPE) is more common for larger, more capable models.
- Transformer architectures are built upon multi-head self-attention, feed-forward networks (MLPs), residual connections, and layer normalization.
- The training loop is crucial, with the objective typically being next-token prediction, often using cross-entropy loss.
- Key parameters for model configuration include vocabulary size, block size (context window), number of layers, attention heads, and embedding dimension.
- Monitoring training involves tracking both training and validation loss to detect overfitting.
- Text generation can be achieved through methods like greedy decoding or temperature sampling, with temperature sampling generally preferred for LLMs to encourage creativity.
- The process can be executed with modest hardware, such as a laptop with 16GB RAM, or more efficiently using free GPUs on Google Colab.
Notable quotes
*The fundamental parts of it are basically haven't changed too much.*
*The way you train it actually makes a big difference in performances.*
*Transformers are commoditized now.*
Unofficial community note. Prefer the recording for nuance.