What is BM25?
BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is.
Checked
BM25 comes out of the probabilistic relevance framework built up in information retrieval research in the 1970s and 1980s. Stephen Robertson and Hugo Zaragoza set out the full derivation in The Probabilistic Relevance Framework: BM25 and Beyond, published in Foundations and Trends in Information Retrieval in December 2009.
It predates embeddings by decades and is still the default similarity in Elasticsearch. In 2026 it also landed inside vector databases, because retrieval for AI agents keeps needing exact terms that embeddings blur.
How the score is built
Term frequency, with saturation
A document that mentions the query term more often scores higher, but each extra mention adds less. The k1 parameter sets how fast that curve flattens. Elasticsearch's default is 1.2.
Inverse document frequency
Rare terms count for more than common ones. A match on a part number that appears in three documents outweighs a match on "invoice", which appears in thousands.
Length normalisation
A long document mentions most words somewhere, so its matches are discounted. The b parameter, default 0.75 in Elasticsearch, sets how strongly length is penalised; 0 turns it off.
Why it still matters next to embeddings
The BEIR benchmark (Thakur et al., April 2021), which tested retrieval models zero-shot across 18 datasets, described BM25 as a robust baseline and found that dense retrieval models, though efficient, often underperformed other approaches on data they were not trained for. Anthropic's Contextual Retrieval study (2024-09-19) found the same pattern in RAG: contextual embeddings alone cut top-20 retrieval failures by 35%, and adding contextual BM25 raised the cut to 49%.
The reason is simple. Identifiers, error codes, clause numbers and product names are exact strings, and BM25 matches exact strings. That is why it forms the keyword half of hybrid search.
Where BM25 fails, and what it is confused with
No sense of meaning
A query about "ending a subscription" scores zero against a passage that only says "cancel the plan". Vector search or learned sparse models cover that gap.
Only as good as the tokenizer
Code identifiers such as getUserById break badly under a plain word tokenizer. Qdrant 1.20 added a code tokenizer that splits on underscores and case changes for exactly this reason.
Scores are not comparable
A BM25 score depends on the query and the corpus statistics, so it cannot be thresholded across queries or added directly to a cosine similarity. Rank-based fusion such as reciprocal rank fusion avoids that problem.
BM25 versus TF-IDF
TF-IDF multiplies raw term counts by rarity. BM25 is a refinement of the same idea that adds term-frequency saturation and length normalisation, which is why it replaced TF-IDF as the default.
Where BM25 ships in 2026
Elasticsearch uses BM25 as its default similarity. On the vector side, turbopuffer shipped its second full-text engine on 2025-12-04, Pinecone made full-text search generally available on 2026-09-09, and Qdrant 1.20 (2026-10-22) added BM25 directly on full-text indexes, with k and b settable per query so tuning needs no reindex. In Postgres, Tiger Data's pg_textsearch 1.0 (2026-03-31) adds BM25 under the PostgreSQL License.
How Motiontic's engineers apply it
Motiontic's engineers run a BM25-only baseline on the client's eval set before adding embeddings, because it shows how much of the retrieval problem is really an exact-match problem. Tokenizer choice and the k1 and b settings are checked against real queries, especially ones full of codes and names, before the keyword side is fused with vectors.
TF-IDF, BM25 and learned sparse retrieval compared
| TF-IDF | BM25 | Learned sparse (SPLADE) | |
|---|---|---|---|
| Term weights from | Counts and rarity | Counts, rarity and length | A trained model |
| Repeated terms | Keep adding score | Saturate (k1) | Weighted by the model |
| Long documents | No correction | Discounted (b) | Handled by the model |
| Synonyms | No | No | Partly, through term expansion |
| Needs a model | No | No | Yes, at index and query time |
Common questions
What is the difference between BM25 and TF-IDF?
Both weight terms by how often they appear and how rare they are. BM25 adds saturation, so the tenth mention of a word adds little, and length normalisation, so long documents do not win by size alone.
Is BM25 still used with LLMs and RAG?
Yes. It is the keyword half of hybrid search. In Anthropic's Contextual Retrieval study, adding BM25 to embeddings moved the cut in top-20 retrieval failures from 35% to 49%.
What do the k1 and b parameters do in BM25?
k1 controls how quickly repeated terms stop adding score, and b controls how much long documents are penalised. Elasticsearch defaults to k1 = 1.2 and b = 0.75.
Can Postgres do BM25 ranking?
Through extensions. Tiger Data's pg_textsearch 1.0, released 2026-03-31 under the PostgreSQL License, adds BM25 ranking that can sit next to pgvector in the same database.
Related
Hybrid search
Keyword ranking and vector similarity, fused into one result list.
Sparse vectors (SPLADE)
Learned keyword weights: the middle ground between BM25 and dense embeddings.
Reciprocal rank fusion (RRF)
Merging keyword and vector results by rank, not by score.
Elasticsearch vs OpenSearch
The original engine and its Apache-licensed fork, compared on licence, governance and vector search.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Sources
- Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (Foundations and Trends in IR, 2009)
- Elastic docs, Similarity settings (BM25 k1 and b defaults)
- Thakur et al., BEIR: a heterogeneous zero-shot retrieval benchmark (arXiv, 2021-04-17)
- Anthropic, Introducing Contextual Retrieval (2024-09-19)
- Qdrant, Qdrant 1.20 release (2026-10-22)
- Tiger Data, pg_textsearch 1.0 (2026-03-31)
- Pinecone, Full-text search generally available (2026-09-09)
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.