World's Fair 2025
Vector Search Benchmark[eting] - Philipp Krenn, Elastic
Overview
This talk addresses the common pitfalls and deceptive practices in vector search benchmarking, often referred to as "benchmarketing." The core thesis is that many published benchmarks are misleading due to biased scenario selection, outdated competitor versions, and a lack of focus on crucial factors like data freshness and query relevance. The speaker emphasizes that truly meaningful benchmarks require rigorous, reproducible, and self-conducted evaluations tailored to specific use cases.
Who should watch
- AI engineers evaluating vector search performance.
- Product managers assessing the viability of vector search solutions.
- Builders seeking to understand the true performance characteristics of vector databases.
- Anyone frustrated by conflicting or unreliable vector search benchmark results.
Key takeaways
- Many benchmarks are designed to favor specific vendors by selecting scenarios that highlight their strengths while ignoring weaknesses.
- Benchmarks often use read-only datasets, which do not reflect real-world workloads that include frequent data updates.
- The impact of filtering on vector search performance, particularly with algorithms like HNSW, is counterintuitive and often misrepresented in benchmarks.
- Outdated competitor versions are sometimes used in benchmarks, making current versions appear less performant than they are.
- Implicit biases related to system architecture and default configurations can lead to skewed benchmark results.
- Cheating can occur by manipulating approximate nearest neighbor search parameters, leading to different result quality without proper reporting of precision and recall.
- Automated and reproducible nightly benchmarks are crucial for tracking performance changes over time and avoiding gradual degradation.
- Users should conduct their own benchmarks with their specific data, workloads, and hardware to obtain accurate performance insights.
Notable quotes
*The problem is that you can have pretty much any product in X or any product in Y.*
*Don't trust anybody's benchmarks and they probably nobody has done exactly the scenario that you want to have.*
*You want to avoid that with your own system that you don't want to boil your own performance.*
Unofficial community note. Prefer the recording for nuance.