All notes
4 min readSearchExplained with Claude
TF-IDF, BM25 and NDCG
BM25 wins on exact matches such as IDs, error codes, names, and rare jargon. Dense retrievers win on paraphrase and meaning.
TF-IDF and BM25
| TF-IDF · idf × raw tf | BM25 · idf × capped tf ÷ length |
|---|---|
| Repeats never plateau — 50 mentions ≠ 50× relevant | Saturation — Controlled by k1 |
| Long docs win by default — More words, more matches | Length normalization — Controlled by b |
raw tf (TF-IDF)BM25, average-length docBM25, this doc
Weight at tf = 5
1.55
Ceiling as tf grows
2.2
This doc vs average-length doc
-13%
- Default to BM25 for lexical search. It's the baseline in Elasticsearch, OpenSearch, Lucene, and most RAG stacks, and plain TF-IDF is now mostly a teaching tool or a cheap feature.
- Use it alongside embeddings, not instead of them. BM25 wins on exact matches such as IDs, error codes, names, and rare jargon. Dense retrievers win on paraphrase and meaning. The current standard is hybrid retrieval (BM25 plus embeddings, merged with reciprocal rank fusion), followed by a reranker.
- The next step up is learned sparse retrieval (SPLADE and similar). It keeps the inverted-index efficiency of BM25 but learns term weights and expansions with a neural model.
- Tuning matters mostly for short or very uneven-length documents. Start with
k1 ≈ 1.2andb ≈ 0.75, then tune on a small labeled set rather than guessing.
NDCG
NDCG scores a ranked list when relevance comes in grades (0 = irrelevant, 3 = perfect) and position matters. It's built in three moves:
- Gain: each result earns points based on its relevance grade, usually 2^rel − 1.
- Discount: divide by log₂(rank + 1). A result at rank 1 counts fully, and the same result at rank 10 counts about 29%.
- Normalize: divide by the DCG of the best possible ordering (IDCG), so scores fall between 0 and 1 and can be compared across queries.
DCG@k = Σ (i = 1 … k) (2^rel_i − 1) / log₂(i + 1)
NDCG@k = DCG@k / IDCG@kReorder the list below and watch the three numbers move:
- 1Doc Arel 11 × 1.00 = 1.00
- 2Doc Brel 37 × 0.63 = 4.42
- 3Doc Crel 00 × 0.50 = 0.00
- 4Doc Drel 23 × 0.43 = 1.29
- 5Doc Erel 00 × 0.39 = 0.00
- 6Doc Frel 3outside top 5
DCG@k
6.71
IDCG@k (best order)
13.35
NDCG@k
0.50
- Tune BM25 with it. NDCG@10 is the standard headline metric (BEIR reports it). Grid-search
k1andbon a small labeled set and keep the pair that maximizes it. - Pair it with recall@k. If the first-stage retriever feeds a reranker or an LLM, recall@50 or @100 tells you whether the relevant documents are in the candidate pool at all. NDCG@10 tells you whether the final order is good.
- IDCG uses all judged documents for the query, not just the ones you retrieved. A relevant document you never surfaced still lowers your score, as it should.
- Unjudged documents count as irrelevant. A new system that surfaces good but unlabeled documents gets penalized unfairly, so check how much of each top-10 is actually judged.
- Libraries differ. Some use linear gain (
rel) and others use2^rel − 1, so numbers aren't comparable across tools until you check. - The trend is LLM-generated relevance labels for building graded test sets cheaply. Calibrate them against a human-labeled sample and check agreement (e.g., Krippendorff's alpha) before trusting NDCG deltas.