85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is recall@k?

Recall@k is a retrieval metric that measures the share of relevant items for a query that appear in the top k results, so a recall@20 of 0.9 means nine in ten relevant passages made the top 20.

Checked

A language model can only use what retrieval hands it. If the passage with the answer is not in the top k, no prompt fixes the result. Recall@k measures exactly that: of everything relevant, how much reached the shortlist.

It is the workhorse metric for RAG retrieval. Anthropic's Contextual Retrieval study (2024-09-19) reports its results as a failure rate defined as 1 minus recall@20, and its headline numbers, a cut from 5.7% to 1.9%, are recall@20 numbers.

Two different recalls

  • Retrieval recall@k

    Measured against human or model relevance labels: did the passages that answer the question appear in the top k? This is what decides answer quality.

  • ANN recall

    Measured against exact search: of the true k nearest vectors, how many did the approximate index return? ANN-Benchmarks defines it as the fraction of true nearest neighbours found, averaged over queries.

  • Why the difference matters

    An index can score 99% ANN recall and still retrieve the wrong passages, because the embeddings put them in the wrong place. Tuning ef_search raises ANN recall; it cannot fix embeddings that rank the wrong passages.

Choosing k

k should match what happens next. If a reranker reads the top 50, recall@50 is the number that matters for the first stage. If the model reads the top 20, recall@20 is the end-to-end measure. Anthropic found passing 20 chunks performed better than 5 or 10 in its tests, which is why its study reports at 20.

Where recall@k misleads, and what it is confused with

  • It ignores order

    A relevant passage at rank 1 and at rank 20 count the same. Pair recall@k with an order-aware metric such as MRR or nDCG when position affects the answer.

  • It needs labels

    Without a golden dataset of questions and their relevant passages, recall cannot be measured. Building that set is most of the work.

  • Chunking changes the unit

    If chunks change, "relevant passage" changes too. Chroma's chunking study measures recall at the token level to compare chunking strategies fairly.

  • Recall@k versus precision@k

    Precision@k asks how much of the top k is relevant. High recall with low precision floods the model with noise; both matter.

  • Recall@k versus pass@k

    pass@k is an agent and code-generation metric: the chance at least one of k attempts succeeds. It measures a different thing despite the similar name.

How Motiontic's engineers apply it

Motiontic's engineers build a labelled set from the client's real questions early, then track recall@k at the reranker's input and at the model's input on every retrieval change. ANN recall is checked separately against exact search, so index tuning and embedding quality are never confused.

Recall@k compared with related metrics

Recall@k compared with related metrics
Recall@kPrecision@kANN recall
Question answeredHow much of what is relevant made the top k?How much of the top k is relevant?How many true nearest neighbours did the index find?
Ground truthRelevance labelsRelevance labelsExact (brute-force) search
Order-awareNoNoNo
Tells a team to fixEmbeddings, chunking, hybrid searchReranking, shortlist sizeIndex parameters

Common questions

What is a good recall@k for RAG?

There is no universal threshold. It depends on k, the corpus and the cost of a miss. Set a target on the system's own eval set and track it across changes.

What is the difference between recall@k and precision@k?

Recall@k measures how many relevant items made the top k. Precision@k measures how many of the top k are relevant. Retrieval for RAG usually needs high recall first, then reranking to raise precision.

How is ANN recall measured?

Run the same queries with exact brute-force search and with the approximate index, then count how many of the true nearest neighbours the index returned.

Is recall@k the same as pass@k?

No. Recall@k is a retrieval metric. pass@k measures whether at least one of k attempts at a task succeeds, and is used for code generation and agents.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.