85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is BM25?

BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is.

Checked

BM25 comes out of the probabilistic relevance framework built up in information retrieval research in the 1970s and 1980s. Stephen Robertson and Hugo Zaragoza set out the full derivation in The Probabilistic Relevance Framework: BM25 and Beyond, published in Foundations and Trends in Information Retrieval in December 2009.

It predates embeddings by decades and is still the default similarity in Elasticsearch. In 2026 it also landed inside vector databases, because retrieval for AI agents keeps needing exact terms that embeddings blur.

How the score is built

  • Term frequency, with saturation

    A document that mentions the query term more often scores higher, but each extra mention adds less. The k1 parameter sets how fast that curve flattens. Elasticsearch's default is 1.2.

  • Inverse document frequency

    Rare terms count for more than common ones. A match on a part number that appears in three documents outweighs a match on "invoice", which appears in thousands.

  • Length normalisation

    A long document mentions most words somewhere, so its matches are discounted. The b parameter, default 0.75 in Elasticsearch, sets how strongly length is penalised; 0 turns it off.

Why it still matters next to embeddings

The BEIR benchmark (Thakur et al., April 2021), which tested retrieval models zero-shot across 18 datasets, described BM25 as a robust baseline and found that dense retrieval models, though efficient, often underperformed other approaches on data they were not trained for. Anthropic's Contextual Retrieval study (2024-09-19) found the same pattern in RAG: contextual embeddings alone cut top-20 retrieval failures by 35%, and adding contextual BM25 raised the cut to 49%.

The reason is simple. Identifiers, error codes, clause numbers and product names are exact strings, and BM25 matches exact strings. That is why it forms the keyword half of hybrid search.

Where BM25 fails, and what it is confused with

  • No sense of meaning

    A query about "ending a subscription" scores zero against a passage that only says "cancel the plan". Vector search or learned sparse models cover that gap.

  • Only as good as the tokenizer

    Code identifiers such as getUserById break badly under a plain word tokenizer. Qdrant 1.20 added a code tokenizer that splits on underscores and case changes for exactly this reason.

  • Scores are not comparable

    A BM25 score depends on the query and the corpus statistics, so it cannot be thresholded across queries or added directly to a cosine similarity. Rank-based fusion such as reciprocal rank fusion avoids that problem.

  • BM25 versus TF-IDF

    TF-IDF multiplies raw term counts by rarity. BM25 is a refinement of the same idea that adds term-frequency saturation and length normalisation, which is why it replaced TF-IDF as the default.

Where BM25 ships in 2026

Elasticsearch uses BM25 as its default similarity. On the vector side, turbopuffer shipped its second full-text engine on 2025-12-04, Pinecone made full-text search generally available on 2026-09-09, and Qdrant 1.20 (2026-10-22) added BM25 directly on full-text indexes, with k and b settable per query so tuning needs no reindex. In Postgres, Tiger Data's pg_textsearch 1.0 (2026-03-31) adds BM25 under the PostgreSQL License.

How Motiontic's engineers apply it

Motiontic's engineers run a BM25-only baseline on the client's eval set before adding embeddings, because it shows how much of the retrieval problem is really an exact-match problem. Tokenizer choice and the k1 and b settings are checked against real queries, especially ones full of codes and names, before the keyword side is fused with vectors.

TF-IDF, BM25 and learned sparse retrieval compared

TF-IDF, BM25 and learned sparse retrieval compared
TF-IDFBM25Learned sparse (SPLADE)
Term weights fromCounts and rarityCounts, rarity and lengthA trained model
Repeated termsKeep adding scoreSaturate (k1)Weighted by the model
Long documentsNo correctionDiscounted (b)Handled by the model
SynonymsNoNoPartly, through term expansion
Needs a modelNoNoYes, at index and query time

Common questions

What is the difference between BM25 and TF-IDF?

Both weight terms by how often they appear and how rare they are. BM25 adds saturation, so the tenth mention of a word adds little, and length normalisation, so long documents do not win by size alone.

Is BM25 still used with LLMs and RAG?

Yes. It is the keyword half of hybrid search. In Anthropic's Contextual Retrieval study, adding BM25 to embeddings moved the cut in top-20 retrieval failures from 35% to 49%.

What do the k1 and b parameters do in BM25?

k1 controls how quickly repeated terms stop adding score, and b controls how much long documents are penalised. Elasticsearch defaults to k1 = 1.2 and b = 0.75.

Can Postgres do BM25 ranking?

Through extensions. Tiger Data's pg_textsearch 1.0, released 2026-03-31 under the PostgreSQL License, adds BM25 ranking that can sit next to pgvector in the same database.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.