What is recall@k?
Recall@k is a retrieval metric that measures the share of relevant items for a query that appear in the top k results, so a recall@20 of 0.9 means nine in ten relevant passages made the top 20.
Checked
A language model can only use what retrieval hands it. If the passage with the answer is not in the top k, no prompt fixes the result. Recall@k measures exactly that: of everything relevant, how much reached the shortlist.
It is the workhorse metric for RAG retrieval. Anthropic's Contextual Retrieval study (2024-09-19) reports its results as a failure rate defined as 1 minus recall@20, and its headline numbers, a cut from 5.7% to 1.9%, are recall@20 numbers.
Two different recalls
Retrieval recall@k
Measured against human or model relevance labels: did the passages that answer the question appear in the top k? This is what decides answer quality.
ANN recall
Measured against exact search: of the true k nearest vectors, how many did the approximate index return? ANN-Benchmarks defines it as the fraction of true nearest neighbours found, averaged over queries.
Why the difference matters
An index can score 99% ANN recall and still retrieve the wrong passages, because the embeddings put them in the wrong place. Tuning ef_search raises ANN recall; it cannot fix embeddings that rank the wrong passages.
Choosing k
k should match what happens next. If a reranker reads the top 50, recall@50 is the number that matters for the first stage. If the model reads the top 20, recall@20 is the end-to-end measure. Anthropic found passing 20 chunks performed better than 5 or 10 in its tests, which is why its study reports at 20.
Where recall@k misleads, and what it is confused with
It ignores order
A relevant passage at rank 1 and at rank 20 count the same. Pair recall@k with an order-aware metric such as MRR or nDCG when position affects the answer.
It needs labels
Without a golden dataset of questions and their relevant passages, recall cannot be measured. Building that set is most of the work.
Chunking changes the unit
If chunks change, "relevant passage" changes too. Chroma's chunking study measures recall at the token level to compare chunking strategies fairly.
Recall@k versus precision@k
Precision@k asks how much of the top k is relevant. High recall with low precision floods the model with noise; both matter.
Recall@k versus pass@k
pass@k is an agent and code-generation metric: the chance at least one of k attempts succeeds. It measures a different thing despite the similar name.
How Motiontic's engineers apply it
Motiontic's engineers build a labelled set from the client's real questions early, then track recall@k at the reranker's input and at the model's input on every retrieval change. ANN recall is checked separately against exact search, so index tuning and embedding quality are never confused.
Recall@k compared with related metrics
| Recall@k | Precision@k | ANN recall | |
|---|---|---|---|
| Question answered | How much of what is relevant made the top k? | How much of the top k is relevant? | How many true nearest neighbours did the index find? |
| Ground truth | Relevance labels | Relevance labels | Exact (brute-force) search |
| Order-aware | No | No | No |
| Tells a team to fix | Embeddings, chunking, hybrid search | Reranking, shortlist size | Index parameters |
Common questions
What is a good recall@k for RAG?
There is no universal threshold. It depends on k, the corpus and the cost of a miss. Set a target on the system's own eval set and track it across changes.
What is the difference between recall@k and precision@k?
Recall@k measures how many relevant items made the top k. Precision@k measures how many of the top k are relevant. Retrieval for RAG usually needs high recall first, then reranking to raise precision.
How is ANN recall measured?
Run the same queries with exact brute-force search and with the approximate index, then count how many of the true nearest neighbours the index returned.
Is recall@k the same as pass@k?
No. Recall@k is a retrieval metric. pass@k measures whether at least one of k attempts at a task succeeds, and is used for code generation and agents.
Related
Golden dataset
The reviewed, versioned test cases an eval trusts as ground truth.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Reranker
A second pass that rereads the shortlist and puts the right passages first.
pass@k and pass^k
Can the agent do it at all, versus does it do it every time.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Sources
- Anthropic, Introducing Contextual Retrieval (2024-09-19)
- ANN-Benchmarks (recall definition)
- Subramanya et al., DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (NeurIPS 2019)
- Chroma, Evaluating Chunking Strategies for Retrieval (2024-07-03)
- Thakur et al., BEIR: a heterogeneous zero-shot retrieval benchmark (arXiv, 2021-04-17)
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.