World's Fair 2026
Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis
Overview
This talk addresses a critical vulnerability in Large Language Models (LLMs): "sleeper agents" or backdoors that remain undetected by standard evaluations. The core thesis is that current defenses, which focus on model behavior or joint feature analysis, are insufficient. The proposed solution lies in analyzing the difference between a base model's activations and a fine-tuned model's activations, a method that reveals these hidden backdoors with high precision.
Who should watch
- AI Engineers
- Product Managers
- Builders working with fine-tuned LLMs
- Those concerned with LLM security and robustness
- Anyone whose LLM passes all current evals but may still have hidden vulnerabilities
Key takeaways
- LLMs can possess "sleeper agents" – backdoors triggered by specific inputs that cause malicious behavior, even if the model passes all standard behavioral tests and evals.
- Traditional defenses like behavioral testing and joint feature interpretability (comparing base and fine-tuned model activations together) are blind to these sleeper agents because the backdoor behavior is only exposed on specific, often unknown, triggers.
- A more effective detection method involves analyzing the difference in activations (delta activations) between a base LLM and its fine-tuned version. This difference highlights directional shifts caused by the backdoor.
- A sparse autoencoder trained on these delta activations (diff SAE) can isolate the backdoor as a distinct feature, achieving a 40x stronger signal compared to joint feature analysis, with perfect precision and zero false positives.
- This detection method is layer-independent, regime-independent (works with full fine-tuning and LoRA), and computationally cheap, making it practical to integrate into existing CI/CD pipelines as a unit test for backdoors.
- The backdoor is essentially a directional shift in the model's activation space, and by subtracting the base model's activations, this specific directional change becomes prominent.
- The fix requires access to the base model checkpoint for comparison; it does not require knowing the specific trigger in advance, as it detects anomalous directional shifts.
- Future work includes developing ensembles for better recall, testing on larger models, exploring adversarial robustness against adaptive attackers, and pairing detection with removal mechanisms.
Notable quotes
*Your LLM deception monitor's broken, the fix is in training data.*
*A model can pass every eval you have and every behavioral monitor you run, it still be carrying a backdoor that flips it into malicious on a trigger you never tested.*
*Backdoors are directions, and the difference is where they live.*
Unofficial community note. Prefer the recording for nuance.