The problem: 4,000 matches, which first?
The agency searches summer shoes and 4,000 of its links match. Nobody reads past the first ten. Which ten? Sorting
by date ignores relevance; sorting by "contains both words" doesn't separate them. A search engine needs a score
for how well each document matches the query. Three ideas build it.
1. Term frequency: more mentions, more relevant
A title that says "shoes" twice is probably more about shoes than one that says it once. So count each query term's occurrences in the document: its term frequency (TF).
The flaw shows straight away: a spammy title that repeats "shoes" five times beats a title that's exactly what the user wants.
2. Inverse document frequency: rare words matter more
In the agency's links, "sale" appears in half of all titles, and "sandals" in very few. Matching "sandals" says much more about a document than matching "sale". Inverse document frequency (IDF) weights each term by how rare it is, typically the logarithm of (number of documents ÷ documents containing the term). TF-IDF multiplies the two.
3. BM25: saturation and length
BM25, the default ranking function in Lucene, Elasticsearch and OpenSearch, keeps IDF and fixes TF in two ways:
- Saturation. Each extra occurrence adds less, and a term's contribution levels off at a ceiling, however often it repeats. Robertson and Zaragoza's review of the framework behind BM25 calls this saturation: one term's contribution can't exceed a limit, however frequently it occurs. The parameter k1 controls how fast it saturates (default 1.2).
- Length normalization. A long document mentions more words just by being long. BM25 compares each document's length with the average and discounts term counts in longer ones. The parameter b controls how strongly (0 = not at all, 1 = fully; default 0.75).
For each query term, BM25 adds:
idf(term) × tf × (k1 + 1) / (tf + k1 × (1 − b + b × length / averageLength))Try it
Terms after analysis: summer, shoe
- 1Shoes, shoes, shoes: the ultimate shoe collection, with shoes for every day of the week and every occasion5.00
- 2Summer sale: shoes and sandals up to 50% off2.00
- 3Summer camp registration for kids1.00
- 4Running shoes buyer's guide1.00
- 5How to clean white shoes1.00
- 6Summer festival tickets1.00
- 7Company summer party photos1.00
- 8Kids' shoes: back-to-school sale1.00
10 titles, average length 4.5 terms after analysis.
Try this
What the lab shows for summer shoes:
| Scoring | First | Second |
|---|---|---|
| Count matches | "Shoes, shoes, shoes…" (5) | "Summer sale: shoes and sandals…" (2) |
| TF-IDF | "Shoes, shoes, shoes…" (3.47) | "Summer sale: shoes and sandals…" (1.61) |
| BM25 (k1 1.2, b 0.75) | "Summer sale: shoes and sandals…" (1.52) | "Shoes, shoes, shoes…" (1.04) |
Counting and TF-IDF both reward repetition. BM25 saturates the five "shoes" and penalises that title for being more
than twice the average length, so the title that matches both words wins. Set k1 to 0 and term frequency stops
counting at all: every match is worth only its IDF. Search sale with b at 0.75 and "Sale ends tonight", the shortest
title, comes first; set b to 0 and all four matches tie.
Beyond BM25
BM25 scores text against text. Real ranking usually combines it with other signals:
- Field weights: a match in the title matters more than one in the notes.
- Business signals: recency, popularity (Snip boosts links with more clicks), the user's own links first.
- Learning to rank: a model trained on what people clicked reorders the top results BM25 found.
- Meaning: BM25 matches words, not ideas; "sneakers" won't find "running shoes". Vector search does (10.05 Vector and hybrid search).
Seen in production
Robertson and Zaragoza
2009BM25
The review traces BM25 from the probabilistic relevance framework. It names saturation, where any one term's contribution to the document's score can't exceed an asymptotic limit however frequently the term occurs, and calls it a very valuable property of BM25's weighting. It also discusses why documents differ in length, for example the "verbosity" hypothesis that some authors simply use more words, which motivates normalizing term frequencies by document length.
What we foundSaturation is the key property: no single term's contribution can exceed a limit, however often it occurs.
The Probabilistic Relevance Framework: BM25 and Beyond (Foundations and Trends in Information Retrieval)staff.city.ac.uk ↗Elastic
DocsElasticsearch similarity
Elasticsearch's default similarity is BM25. Its k1 parameter controls non-linear term frequency normalization, or saturation, defaulting to 1.2, and b controls how much document length normalizes term frequencies, defaulting to 0.75. Elastic's Practical BM25 series explains the variables in more depth.
What we foundBM25 is the default, with k1 = 1.2 for saturation and b = 0.75 for length normalization.
Similarity settings (Elasticsearch documentation)elastic.co ↗Snip indexes each link's title and its full destination page text in one field. Searches for 'pricing' now rank huge product pages above links titled 'Pricing'. Which change helps most?
Recap
Interview questions
How do search engines rank results? Explain TF-IDF and BM25.