World's Fair 2025
The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger
Overview
This talk argues that Large Language Models (LLMs) excel due to coherence, not intelligence. While LLMs can produce outputs that feel remarkably insightful and intelligent, they lack genuine understanding, intent, or desire. The speaker proposes that coherence is a system property, not a cognitive one, and explains how LLMs construct meaning on demand through pattern alignment within a high-dimensional latent space, rather than retrieving stored knowledge.
Who should watch
- AI engineers and builders exploring LLM capabilities beyond simple text generation.
- Product Managers and UX designers seeking to understand the underlying mechanisms that make LLMs feel useful.
- Anyone frustrated by LLM brittleness or prompt sensitivity and looking for a theoretical framework to improve interactions.
- Researchers interested in the emergent properties of LLMs and the distinction between coherence and intelligence.
Key takeaways
- LLMs create the *feeling* of intelligence by generating coherent, consistent, and stable outputs that align with user prompts, rather than by thinking or understanding.
- The "coherence trap" describes how LLMs can appear smart because their outputs align with user expectations and context, even without true comprehension.
- A practical framework for interacting with LLMs involves a "frame, generate, judge, iterate" loop, emphasizing evaluation and refinement of outputs.
- Hallucinations in LLMs are not necessarily bugs but emergent behaviors indicating the model's attempt to complete patterns and maintain coherence when information is insufficient.
- Retrieval Augmented Generation (RAG) acts as factual anchors, providing structural scaffolding to guide the LLM's coherence and steer outputs toward reality.
- LLMs operate in a high-dimensional latent space where prompts act as force vectors, activating and merging specialized patterns to construct meaning on demand.
- Understanding LLMs as coherent systems, rather than intelligent agents, shifts the focus from control to designing for structured resonance and collaborative interaction.
- Key properties of coherence include relevance, consistency, and stability, allowing models to withstand interrogation and maintain a unified output.
Notable quotes
*LLMs are not intelligent. They're coherent. And that's their superpower.*
*The magic happens in the collaborative dance. So stop chasing intelligence and start designing for structured residents.*
Unofficial community note. Prefer the recording for nuance.