World's Fair 2025
New York Times' Connections: A Case Study on NLP in Word Games — Shafik Quoraishee, NYT Games
Overview
This talk explores the New York Times Connections word game as a case study for evaluating AI's abstract reasoning capabilities. The presenter, a game developer at the NYT Games team, details independent research into how AI can approach solving the game, which involves grouping 16 words into four sets of four related terms. The research investigates the game's potential as a benchmarking tool, highlighting how its intentional decoys and need for explainable reasoning challenge current AI models.
Who should watch
- AI engineers interested in evaluating abstract reasoning and problem-solving in LLMs.
- Product Managers and builders looking for novel applications of NLP and AI in games.
- Developers seeking to understand how to model complex word relationships for AI.
- Anyone curious about the intersection of word games, AI, and computational linguistics.
Key takeaways
- The Connections game, with its distinct difficulty levels (yellow, green, blue, purple) and decoy words, serves as a challenging testbed for AI's ability to avoid superficial similarities and perform abstract reasoning.
- Human problem-solving in Connections often involves a combination of System 1 (intuitive, fast) and System 2 (deliberate, slow) thinking, a duality that AI models struggle to replicate perfectly.
- The game can be modeled as an augmented graph coloring problem, where words are vertices, categories are colors, and the strength of connection is represented by edges.
- Semantic similarity alone is insufficient for AI to solve Connections; understanding multiple types of relationships (orthography, morphology, encyclopedic, associative) and their alignment is crucial.
- Relational alignment scores can computationally differentiate between easy and hard puzzles, offering a metric for puzzle difficulty.
- Advanced techniques like multi-dimensional hypergraphs and graph neural networks (GNNs) combined with reinforcement learning are being explored to build more sophisticated AI solvers.
- A key motivation for this research is to develop more transparent and explainable AI solutions, moving beyond the "black box" nature of some LLM outputs and addressing concerns about LLMs potentially just retrieving existing internet solutions.
Notable quotes
*The game's intentional decoys test whether AI can over avoid overfitting or superficial similarities.*
*Semantic similarity is not all you need.*
*I'm actually trying to increase this intelligence space like procedurally so that you can have a trackable and explainable way. You know, explainable AI, that's what I'm all about.*
Unofficial community note. Prefer the recording for nuance.