Europe 2026
$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpentero
Overview
This talk addresses the escalating sophistication of AI attacks, moving beyond simple prompt injection to complex exploits targeting LLM interfaces, data, and even internal mechanisms. It proposes a practical, low-latency, self-hosted defensive layer built by fine-tuning a modern encoder model, specifically ModernBERT, to act as a guardrail against these threats. The approach emphasizes efficiency and cost-effectiveness, aiming for a defense under a dollar per instance.
Who should watch
- AI Engineers
- Product Managers
- Builders working with LLMs
- Those concerned with AI security and safety
- Individuals looking to implement defensive layers for AI systems
Key takeaways
- AI attacks have evolved from basic prompt injection to sophisticated methods exploiting LLM interfaces, retrieval-augmented generation (RAG), tools, and model internals.
- A core challenge is the lack of native separation of concerns within LLMs between system instructions and user data, making them vulnerable.
- Fine-tuning encoder models like ModernBERT offers an efficient and low-latency solution for implementing AI safety checks, with a demonstrated inference time of around 35 milliseconds.
- ModernBERT's architectural improvements, including alternating attention, unpadding and sequence packing, deep and narrow design, and rotary positional encoding, contribute to its efficiency and effectiveness.
- Flash attention is crucial for optimizing memory transfers on GPUs, significantly reducing latency and enabling faster processing.
- The proposed defensive layer can be self-hosted, enhancing privacy and potentially reducing costs compared to external API calls.
- The Inject Guard dataset, comprising 75,000 labeled examples, is used for fine-tuning, achieving approximately 85% accuracy with the ModernBERT large version.
- AI safety is a shared responsibility, and building effective defensive layers is achievable even with commodity hardware.
Notable quotes
*The LLMs, they have no native separation of concerns between the system controls and the data.*
*We're not building defensive layers to pass a security audit. We have to build safety mechanisms that protect machines, human and humans and society.*
Unofficial community note. Prefer the recording for nuance.