World's Fair 2024
Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley
Overview
This talk focuses on Snorkel AI's approach to data development for enterprise AI models, emphasizing the challenges of achieving enterprise-grade quality, latency, and cost requirements with off-the-shelf large language models. Snorkel AI specializes in developing data to fine-tune these models, enabling them to meet specific business needs. The discussion also touches upon the infrastructure and best practices for running large-scale AI training and inference workloads on Azure.
Who should watch
- AI Engineers
- Product Managers
- Builders working on custom AI solutions
- Those facing challenges with LLM quality and cost for enterprise deployment
- Individuals interested in AI infrastructure and distributed training
Key takeaways
- Out-of-the-box LLMs often fail to meet enterprise quality, latency, and cost requirements, necessitating custom data development for fine-tuning.
- Data development is challenging, with RAG being a starting point but not a complete solution; data quality and relevance are paramount.
- Snorkel AI, originating from Stanford AI Lab, focuses on making data development programmatic, scalable, and auditable, while keeping subject matter experts in the loop.
- Key research areas include fine-grained evaluation for long-context models, enterprise alignment with regulations, and multimodal alignment using large vision-language models for synthetic data generation.
- Running large-scale distributed training on Azure involves frameworks like PyTorch and Horovod, utilizing Azure VMs and a shared Network File System (NFS).
- Architectural considerations for distributed training include balancing node count for batch size, mitigating networking bottlenecks, and ensuring efficient data reading from NFS.
- GPU utilization is a critical metric for identifying bottlenecks, with `nvidia-smi` being a valuable tool for debugging.
- Azure's infrastructure offers advantages in availability, reliability, NFS throughput, and ease of use, supporting flexible compute capacity.
Notable quotes
*Out of the box LLMs rarely meet enterprise quality latency and cost requirements.*
*RAG is just a starting point.*
*Make data development programmatic, scalable and auditable.*
Unofficial community note. Prefer the recording for nuance.