Europe 2026
Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA
Overview
This talk explores the practical performance of running large language models (LLMs) locally on the NVIDIA Jetson Spark, a system designed for AI development. It addresses challenges like memory limitations and software stack access that often push AI workloads to the cloud. The presentation emphasizes how local solutions can enhance developer productivity by bringing AI development closer to the user, enabling faster iteration and addressing concerns like cost predictability and data residency.
Who should watch
- AI Engineers
- Product Managers
- Developers building and deploying AI systems
- Those facing cloud dependency for AI development
- Individuals concerned with cost, data residency, and latency in AI workflows
- Anyone looking to improve developer productivity with local AI solutions
Key takeaways
- The NVIDIA Jetson Spark, powered by the GB10 Grace Blackwell superchip with 128 GB of unified memory, can run models up to approximately 200 billion parameters locally.
- It utilizes the same NVIDIA AI software stack as production environments, facilitating seamless workflow transitions from desktop to data center or cloud.
- An automated benchmarking harness was developed to ensure rigorous and reproducible performance measurements, including environment isolation, warm-up runs, and metric logging.
- Time to first token is a critical metric for user-perceived performance, indicating application responsiveness.
- Quantization, specifically NVFB4 4-bit floating-point, significantly impacts performance, allowing larger models to achieve higher throughput and faster time to first token.
- A 14 billion parameter NVFB4 model achieved 20.19 tokens per second and was 3.4 times faster to first token compared to the unoptimized 14 billion base model.
- Memory capacity is distinct from memory bandwidth; while the Jetson Spark has ample memory, efficient data movement is crucial for throughput.
- The Jetson Spark is ideal for steady-state workloads, privacy-sensitive data, and rapid prototyping, enabling local iteration before scaling to larger environments.
Notable quotes
*The key idea here is not replacing the cloud, but bringing powerful AI development closer to the developer.*
*By leveraging Nvidia's NVFB4 4-bit floating-point quantization, we were able to maintain a sophisticated, high-intelligent model at a throughput that is still faster than the average human reading speed.*
*A key takeaway here from this data is that memory capacity is not the same as the memory bandwidth.*
Unofficial community note. Prefer the recording for nuance.