Europe 2026

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

Filip Makraduli , Superlinked

Overview

This talk addresses the gap in infrastructure for small model inference, particularly for AI search and document processing. The speaker, Filip Makraduli, shares his journey from overlooking inference to co-founding the open-source Superlinked Inference Engine (SIE). The core thesis is that effective inference for agentic workflows requires a holistic approach combining robust model support with scalable infrastructure, enabling developers to efficiently deploy and manage diverse small models.

Who should watch

• AI Engineers
• Product Managers
• Builders working with AI agents and workflows
• Those facing challenges with context rot in AI applications
• Developers looking for efficient small model inference solutions
• Individuals interested in open-source infrastructure for AI

Key takeaways

• Small models are crucial for managing context in agentic workflows, pre-processing data, and enabling efficient tool calling.
• Inference for small models is not solely about adding more GPUs; it requires efficient memory utilization and the ability to hot-swap models on a single GPU to reduce costs and increase utilization.
• A comprehensive inference solution needs to go beyond just model serving to include production-ready features like routing, auto-scaling, queuing, and monitoring.
• Supporting a wide range of open-source models is essential, but requires adapting inference engines to handle architectural differences, such as varying attention mechanisms and positional embeddings.
• The Superlinked Inference Engine (SIE) offers an open-source solution that combines adaptable model inference with scalable infrastructure, aiming to provide an end-to-end experience.
• SIE's infrastructure layer includes primitives for encoding, scoring, and extraction, along with routing, queuing, and auto-scaling capabilities using tools like KEDA and Prometheus.
• The project emphasizes supporting diverse model architectures by re-implementing forward passes and enabling variable-length flash attention to prevent wasted compute.
• The solution is deployable via Terraform, Helm charts, and Docker images, simplifying the process of setting up and managing inference clusters.

Notable quotes

Watch on YouTube →

Unofficial community note. Prefer the recording for nuance.