BM25F from Scratch (softwaredoug.com)

This post walks through BM25 from first principles and builds BM25F (multi‑field BM25) step by step. It breaks relevance into three core signals—term frequency (TF), document frequency (DF) and field length—and shows how BM25 encodes human relevance judgments via three simple transforms: a log‑style IDF (idf = log(1 + (N − df + 0.5)/(df + 0.5))), length normalization with parameter b (scaled_tf ≈ tf / (1 − b + b * doc_len / avg_doc_len)), and term‑frequency saturation controlled by k1 (bm25_tf ≈ tf / (tf + k1)). Combining these yields the classic BM25 TF*IDF where TF is length‑aware and saturating. The article then diagnoses why naive per‑field scoring breaks (e.g., rare words in short fields like title getting outsized weight) and why simply summing field BM25 scores double‑counts the early, high‑leverage portions of each field’s TF curve. BM25F’s practical fix: blend document frequencies across fields (Elasticsearch’s cross_fields uses max DF) to compute a single IDF, length‑normalize each field’s TF, sum those scaled TFs, then apply BM25 saturation to the combined TF. Final score = blended_idf * bm25_tf(sum_scaled_tfs). The writeup highlights implications for sparse retrieval and multi‑field sparse transformer models, and flags tokenization, DF estimation and per‑field tuning (b, k1, average lengths) as key knobs for improving multi‑field lexical retrieval.