Chapter 18: Classical Information Retrieval (IR) You Are Expected to Know

00-START-HERE · 01-READING-ROUTES · 17_operating_a_vector_store · 19_learning_to_rank This chapter prepares you to explain how classical lexical ranking supports a Retrieval-Augmented Generation (RAG) system, where its assumptions fail, and why sparse retrieval still protects rare queries.

TL;DR

• Boolean queries enforce exact logical membership, but do not themselves rank the matching set. AND can starve a query; OR can flood it. Expected selectivity depends on term correlations.
• TF-IDF coordinates reward repetition; unnormalized linear and log-TF weights have no ceiling. Cosine is bounded and invariant to pure positive vector rescaling, but can penalize useful breadth under the specified section model.
• BM25 bounds each positive term contribution and gives document length a tunable role through $k_1$ and $b$. Its shape is a tractable model, not a guarantee of answer relevance.
• Zero-feedback RSJ yields the classic, possibly negative IDF. Guarded BM25 adds a different transformation that keeps term contributions positive; do not conflate their derivations.
• BIM assumes binary term incidence and conditional term independence. BM25 adds TF saturation and length normalization without explicitly modeling phrases or joint context utility.
• Exact sparse fields can preserve rare identifiers. Mean pooling assigns token coefficients, not a fixed percentage of vector direction or learned importance; dense and learned-sparse performance must be measured.
• Keep authorization and other mandatory validity rules strict. Evaluate sparse, dense and hybrid retrieval by query type, freshness, length, cost and failure severity.

The story

Imagine one librarian serving a large policy archive. A visitor asks whether the retention policy permits deleting customer records after eighteen months. In this illustration the librarian requires every clause in retention AND deletion AND customer AND gdpr, and no folder passes. The worked count model below explains why an empty result can be likely, not why it is certain for every real archive. She switches to OR, accepting a folder that contains any of the words. Under the stated frequency model the expected matching set is about $1.642$ million folders. Boolean membership provides no relevance order; a particular engine may add one. The strict gate starved her and the loose gate flooded her. The librarian gives each folder a graded score. An extended Boolean rule uses exponent $p\ge1$ to trade strong partial coverage against weaker coverage of every term. At $p=1$, soft AND and soft OR are the same mean. As $p\to\infty$, they approach min and max; these recover hard Boolean membership only for binary weights. Next she tries TF-IDF: a term receives within-document frequency and corpus-rarity weights. An unnormalized linear-TF coordinate pays the same increment for every occurrence. Cosine normalization changes the final marginal score, so the coordinate-level repetition problem must not be described as an unbounded cosine score. Cosine divides by vector lengths. Pure positive rescaling cancels exactly for fixed linear-TF vectors. Under a more restrictive model—ten mutually orthogonal, equal-norm sections, only one contributing to the query—the whole guideline scores $1/\sqrt{10}\approx31.6228\%$ of that section alone. Real sections need not satisfy those assumptions. The librarian adopts BM25. At average length the first occurrence has TF factor $1$, and the limiting factor is $k_1+1$. With length and statistics held fixed, all later occurrences share a limiting additional budget $k_1$; no finite count attains that limit. BM25 also has a length dial $b$. At $b=0$ the TF factor ignores relative length. At $b=1$ the effective saturation constant is multiplied by relative length, which is not the same as dividing the final score directly by length. Intermediate settings let the librarian tune the verbosity-versus-breadth trade-off. Her audit notebook starts with the Probability Ranking Principle: under additive expected gain and the stated costs, rank by marginal relevance probability. Independent document labels are not required. A two-by-two table then estimates a term log odds ratio. With zero feedback it yields raw RSJ IDF; the positive guarded BM25 form is a further modeling choice. The notebook treats incidence as binary and assumes terms independent conditional on relevance and query. Its unexpanded BIM score cannot credit a non-query term. BM25 adds repeated-term and length evidence, but not explicit phrase interactions. Query expansion, phrase fields, learned sparse models, late interaction and cross-encoders offer different ways to add that missing evidence. Finally a visitor asks about ERR_OSSL_EVP_UNSUPPORTED. If compatible exact-field analyzers retain that string intact, sparse retrieval can give it its own coordinate and a guarded IDF of approximately $11.982930$ in the worked collection. A hypothetical mean-pooled encoder gives ten identifier positions out of $180$ positions total coefficient mass $10/180\approx5.5556\%$. That is not a percentage of vector direction, importance or retrieval probability, and it is not a statement about every DPR encoder. The librarian keeps both catalogs when evaluation justifies the operational cost. Sparse matching often helps exact terms; dense representations often help paraphrases. Neither advantage is universal. Freshness also depends on ingestion, analyzers, refresh and replication. Mandatory policy gates remain independent of both scores.

Decoder table

Core mechanics

Mathematical conventions

Let $\mathcal V$ be the indexed vocabulary, $V=\lvert\mathcal V\rvert$, $N$ the document count, $T(d)$ the terms in document $d$, and $Q(q)$ the set of distinct query terms, with $n=\lvert Q(q)\rvert\ge1$. Query $q$ and document $d$ also denote their vectors when a vector formula is used. Write $d_{\mathrm{emb}}$ for dense-vector dimension so that it is not confused with a document. Document frequency is $n_t=\mathop{\mathrm{df}}\nolimits(t)$ and within-document count is $f_{t,d}\ge0$. Counts, word lengths and model-token lengths are not interchangeable; BM25 lengths must use one consistent indexed-token convention. All logarithms are natural. Distinguish three weights: classical TF-IDF $I_{\mathrm{TF}}(t)=\ln(N/n_t)$ for $1\le n_t\le N$; zero-feedback raw RSJ $w_t^{(0)}=\ln((N-n_t+0.5)/(n_t+0.5))$; and guarded BM25 $I_{\mathrm B}(t)=\ln(1+(N-n_t+0.5)/(n_t+0.5))$. The last two are not equal. Log odds ratios are dimensionless, conventionally reported in nats, not probabilities. The clinical TF-IDF example uses the same IDF-weighted query throughout: $q_t=I_{\mathrm{TF}}(t)\mathbf1[t\in Q(q)]$ and $d_t=g(f_{t,d})I_{\mathrm{TF}}(t)$. Linear TF uses $g(f)=f$; sublinear TF uses $g(0)=0$ and $g(f)=1+\ln f$ for $f\ge1$. Thus its unnormalized dot product is $\sum_{t\in Q(q)}g(f_{t,d})I_{\mathrm{TF}}(t)^2$. The release-note example separately names a single-IDF linear lexical baseline with unit query indicators; it is not this symmetric TF-IDF dot product. Cosine requires nonzero vectors; define a zero-vector fallback rather than divide by zero. For BM25, use positive document length $L_d$, positive mean $\overline L$, relative length $\ell_d=L_d/\overline L$, $k_1\gt 0$ and $b\in[0,1]$. Each distinct query term occurs once in the score sum; extra query-frequency factors, field-specific parameters and boosts are outside the displayed baseline. A TF saturation curve holds relative length and collection statistics fixed. Literally appending text can also change length and collection statistics. Expected Boolean counts require an explicit term-occurrence model. An empty-result probability additionally needs a model across documents. Probability-ranking optimality below concerns additive expected utility; it does not require independent relevance labels across documents. BIM instead assumes term independence conditional on relevance and query. Distinguish these assumptions. Storage uses $1\,\mathrm{GB}=10^9\,\mathrm B$ and fp32 payload $4\,\mathrm{B/dimension}$. Posting and vector payload estimates exclude the stated index and runtime overhead. Keep full precision in calculations, round only reported values, and use $\approx$ for approximations. Software/version claims, source vignettes and benchmark results remain reported inputs, not outcomes proved by the arithmetic.

18.1 Boolean and extended Boolean models

Boolean retrieval defines a set

• Model: $T(d)\subseteq\mathcal V$ is the document term set. Document $d$ matches when that set satisfies the logical query $q$.
• Role: metadata filters, must and must_not can impose mandatory rules. Extracted filters also require validation; a logical operator does not make its input correct.
• Limitation: Boolean membership returns a set, not an intrinsic relevance score. An engine may rank that set separately.
• Cost: merging two sorted lists gives a basic AND or OR algorithm in $O(\lvert P_1\rvert+\lvert P_2\rvert)$ time. NOT is relative to a candidate universe and needs a difference operation. Skips, bitmaps, compression and dynamic pruning change actual work.

Conjunction shrinks geometrically

For a representative document, assume the $n$ query-term presence events are independent with probabilities $n_i/N$. Linearity of expectation then gives the count formula below. It does not require document-to-document independence, but corpus marginals alone do not justify the within-document product.

• Interpretation: under this model each additional conjunct multiplies the expected count by $n_i/N$. Correlated terms require joint or conditional occurrence probabilities instead.
• Limitation: fewer matches do not establish higher precision or correct user intent. OR expands membership but supplies no relevance ranking by itself.
• Failure mode: product-like shrinkage can create feast or famine. The count formula is a planning model, not a claim that a particular result count is fractional or deterministically zero.

A fallback ladder does not create one ranking

• A fixed ladder can execute the full conjunction, then remove a selected optional term at each rung. A four-term ladder can have four query executions; it does not have to enumerate every subset.
• There are $2^n$ possible term subsets, but that is the unrestricted search space, not the cost of one fixed ladder. Caching, reusable iterators and skips may reduce repeated work.
• Different rungs do not automatically share a relevance score. Explicit rung priority or a common scorer can combine them; calling the sets inherently incomparable is too strong.
• Dropping a term guesses which part of intent is dispensable. Never relax authorization or another mandatory validity condition merely to obtain results.

Extended Boolean retrieval softens the operator

For $n\ge1$, equal query weights, normalized document-term weights $w_i(d)\in[0,1]$ and $1\le p\lt \infty$, define the following graded operators. They are scores, not authorization checks.

• Soft AND measures normalized distance from the all-present corner; soft OR measures normalized distance from the origin. Both lie in $[0,1]$.
• Exponent $p$ controls the trade-off between coverage and strong partial matches. Finite-$p$ soft AND can be positive when a term is missing, so it is not a strict conjunction.
• The source attributes better results at intermediate values, notably $p=2$, to Salton, Fox and Wu (1983). That is a reported experiment on their collections, not a universal optimal exponent.

The endpoints explain the model

At $p=1$, the two equal-weight operators are exactly the same arithmetic mean:

At $p=1$, these operators no longer distinguish AND from OR. This is valid as a named soft coordination score, but not as a promise of hard AND. As $p\to\infty$, $S_{\mathrm{AND},p}\to\min_i w_i$ and $S_{\mathrm{OR},p}\to\max_i w_i$. On binary weights, those endpoints recover Boolean membership; on graded weights they are fuzzy min/max scores.

The compliance query makes the cliff concrete

• Inputs: $N=10^7$ chunks and frequencies $1.5\times10^6$, $1.2\times10^5$, $4\times10^4$ and $8\times10^3$ for customer, deletion, retention and gdpr. Their marginal shares are $0.15$, $0.012$, $0.004$ and $0.0008$.
• Independent-term model: expected conjunction counts are $1{,}500{,}000\to18{,}000\to72\to0.0576$. Adding gdpr multiplies the previous expectation by $0.0008=1/1{,}250$. Do not round $0.0576$ into a guaranteed zero.
• If document match indicators are also IID, $\lvert A_{\wedge}\rvert\sim\mathop{\mathrm{Binomial}}\nolimits(10^7,5.76\times10^{-9})$ and $\Pr(\lvert A_{\wedge}\rvert=0)=(1-5.76\times10^{-9})^{10^7}\approx0.944027483$. Thus an empty result is approximately $94.4027\%$ likely in that additional model. From the expectation alone, Markov's inequality gives $\Pr(\lvert A_{\wedge}\rvert\ge1)\le0.0576$ without document independence.
• Under the same within-document term-independence model, expected OR membership is $N[1-(0.85)(0.988)(0.996)(0.9992)]=1{,}642{,}283.5264\approx1.642284\times10^6$. An actual count is integral. Real term correlations can change both estimates.

The worked soft score exposes the exchange rate

Let document A have $(0.9,0.8,0.6,0)$ and B have $(0.5,0.5,0.5,0.5)$. At $p=2$:

A scores $0.575$ at $p=1$, $0.45$ at $p=2$, and tends to $0$ as $p\to\infty$. B scores $0.5$ for every $p\ge1$, since all its term weights and complements are equal. This is a change in ranking preference, not a change in the documents.

Soft scoring costs more postings work

• A straightforward graded scorer evaluates the union of matching documents rather than just the hard intersection. That can cost more postings work, but one graded score avoids an implicit unscored fallback ladder.
• Full-list accounting here is $1{,}500{,}000+120{,}000+40{,}000+8{,}000=1{,}668{,}000$ postings, not $1{,}668{,}000$ distinct candidates.
• Stipulating $1\,\mathrm{ns/posting}$ gives $1.668\,\mathrm{ms}$ of serial posting work. This excludes decoding, score arithmetic, heaps, memory stalls and overhead; it is not a measured latency or proof of a hundredfold gap against hard AND.
• Selective hard-first evaluation, skips, block bounds and dynamic pruning can avoid full decoding when their invariants hold. The actual ratio requires an implementation and workload measurement.

Practical Boolean decisions

• Keep permissions, tenant boundaries, legal holds and other mandatory validity rules strict. A per-query leak probability $10^{-4}$ over $10^6$ queries implies $100$ expected leak events, not a guaranteed daily count or a count of distinct people. Fail closed and validate every serving path; widening a pool cannot repair unauthorized admission.
• Recency or source preference can be soft only when it is genuinely optional. An explicit date, jurisdiction or version requirement may instead be mandatory.
• Three of four optional clauses is $75\%$. The source's Solr string 2<-1 5<-2 6<90% is a conditional staircase, not globally all-but-one: under that syntax, up to two clauses require all; three through five require all but one; six require all but two; above six require the integer percentage rule. Check the current parser and rounding semantics.
• Use exact fields or $100\%$ matching when every identifier component is necessary. Preserve the query/document analyzer contract.
• Start a soft-scoring experiment at $p=2$ if useful, then validate by query type. Codes may be better served by exact matching than by simply raising $p$.
• Bucket zero-result rate by query length and intent: an aggregate $3\%$ can coexist with $40\%$ for five-term queries. Their coexistence also depends on cohort weights.
• A suggested cap of two machine-extracted optional conjuncts is a workload heuristic, not a correctness theorem. Never discard a necessary security or validity clause to satisfy that cap.
• An offline OR probe can help distinguish an over-restrictive conjunction from analyzer, vocabulary or corpus-coverage problems. OR hits alone do not prove relevant answers or correct intent. Use explicit, evaluated broadening rather than silent term dropping.
• Keep mandatory negation strict. Soft NOT makes an exclusion a preference, not a guarantee.

18.2 Vector space model and TF-IDF’s two failures

Vector space scoring creates an order

Salton, Wong and Yang (1975) are the source's attribution for the vocabulary-space model. For nonzero query and document vectors, cosine similarity is:

• A score orders candidates up to ties; it is not automatically a total order. Add a deterministic tie policy if one is required.
• Cosine lies in $[-1,1]$ for general real vectors, and in $[0,1]$ for the nonnegative TF-IDF vectors here. Zero vectors need an explicit fallback.
• Using angle as a relevance proxy is a modeling choice, not a probability guarantee. Sparse dot-product work depends on postings, intersections and the scoring implementation.

Raw counts need rarity and length corrections

The classical weight used here is $I_t=I_{\mathrm{TF}}(t)=\ln(N/n_t)$ for $1\le n_t\le N$. A document coordinate is $g(f_{t,d})I_t$ and the clinical query coordinate is $I_t$ for each query term. Hence the query-term dot product uses $I_t^2$. A unit-weight query would give a different, single-IDF score; mixing the two conventions changes the arithmetic. Corpus rarity and vector normalization address different effects, but neither certifies relevance.

Failure one is linear repetition

Holding IDF fixed, the derivative of a linear-TF document coordinate is constant. Under this chapter's IDF-weighted clinical query, the corresponding unnormalized dot-product derivative has another IDF factor:

For integer counts these derivatives correspond to constant coordinate or dot-product increments. They do not describe the derivative of normalized cosine, whose denominator also changes. Repetition can distort a lexical relevance proxy, but the existence and severity of a rank reversal depend on the other coordinates and the chosen normalization.

Failure two confuses scope with verbosity

For fixed linear-TF weights and $\alpha\gt 0$, $\mathop{\mathrm{sim}}\nolimits(q,\alpha d)=\mathop{\mathrm{sim}}\nolimits(q,d)$. This is invariance to rescaling the whole vector, not to repeating only one query term, and raw-text duplication under log TF is not generally a pure rescaling. To isolate breadth, let $d=\sum_{i=1}^m c_i$ with mutually orthogonal, equal nonzero norms and $q^{\mathsf T}c_i=0$ for $i\gt 1$. Then:

For $m=10$, the factor is $1/\sqrt{10}\approx0.316227766$, a relative decrease of approximately $68.3772\%$. Overlapping, unequal-norm or additionally query-relevant sections change it. Ordinary cosine has no explicit tunable breadth parameter, but it is still bounded; neither this example nor vector-rescaling invariance proves that arbitrary keyword stuffing is free.

Log damping helps but never caps repetition

Sublinear TF is $g(0)=0$ and $g(f)=1+\ln f$ for $f\ge1$. It reduces the unnormalized clinical A-to-B dot-product ratio from approximately $4.105133$ to $1.372225$. With $1{,}000$ insulin occurrences the ratio is approximately $3.113775$. The log-TF coordinate is unbounded as a mathematical function, but normalized cosine is not. The source's scikit-learn example uses sublinear_tf=True; an absent term must not be passed to $\ln0$.

Default TF-IDF still contains both defects

The source describes TfidfVectorizer defaults sublinear_tf=False and norm='l2'. Verify the installed version and analyzer. Its usual smooth_idf=True formula is $I_t^{\mathrm{smooth}}=\ln((1+N)/(1+n_t))+1$; smooth_idf is a Boolean option, not the formula's value. These defaults change the exact weights from the classical unsmoothed clinical example. Linear coordinate growth and the conditional breadth effect remain distinct from an unbounded normalized score.

The clinical example prices stuffing

• Inputs: $N=10^6$ and frequencies $10^4$, $10^5$, $5\times10^4$ for insulin, dosage and pediatric. Exact IDFs are $I_i=\ln100$, $I_d=\ln10$ and $I_p=\ln20$, approximately $4.605170186$, $2.302585093$ and $2.995732274$.
• A is a $1{,}200$-word page containing insulin $12$ times and neither other query term. B is a $180$-word table with counts $(2,2,1)$. Other terms do not contribute to the query dot product, but will contribute to cosine norms.
• Use the IDF-weighted query defined above. Linear-TF dot products are $s_A=12I_i^2\approx254.491109$ and $s_B=2I_i^2+2I_d^2+I_p^2\approx61.993393$.
• Thus $s_A/s_B\approx4.105133$. These are unnormalized vector scores, not relevance probabilities or cosine scores. A unit-weight query would produce different numbers and must not be silently substituted.

Sublinear and cosine configurations still rank badly

• With the same query and log-TF document coordinates, $s_A=(1+\ln12)I_i^2\approx73.906480$ and $s_B=(1+\ln2)(I_i^2+I_d^2)+I_p^2\approx53.858881$.
• Their ratio is approximately $1.372225$. This isolates TF damping before any vector normalization.
• At $f_{i,A}=1{,}000$, $s_A=(1+\ln1{,}000)I_i^2\approx167.704451$, or approximately $3.113775$ times B. This is a mathematical counterexample for the unnormalized score, not an assertion about the final cosine ranking.

Cosine promotes the tiny heading

• Keep that same IDF-weighted query and log-TF document representation. The query norm is $\lVert q\rVert_2=\sqrt{I_i^2+I_d^2+I_p^2}\approx5.956837$.
• Let heading H contain insulin and dosage once each and no other indexed terms. Its norm is $\sqrt{I_i^2+I_d^2}\approx5.148737$, and $q^{\mathsf T}H=I_i^2+I_d^2\approx26.509491$. Therefore $\mathop{\mathrm{sim}}\nolimits(q,H)\approx0.864340778$.
• Stipulate that B's non-query coordinates contribute exactly $220$ to its squared weighted norm. This is a toy norm budget, not a fact inferred from its word count. Then $\lVert B\rVert_2=\sqrt{(1+\ln2)^2(I_i^2+I_d^2)+I_p^2+220}\approx17.463402$, and $\mathop{\mathrm{sim}}\nolimits(q,B)=\frac{53.858881\ldots}{(5.956836\ldots)(17.463402\ldots)}\approx0.517741260$.
• H's cosine is approximately $1.669445$ times B's. If B is the sole query-contributing section among ten orthogonal equal-norm sections, the whole-document score is $0.517741260\ldots/\sqrt{10}\approx0.163724162$. The norm budget and section assumptions drive this illustrative ranking; no answer-quality guarantee follows.

BM25 supplies a tunable alternative

At $k_1=1.2$, $b=0.75$, relative length $\ell=10$ and $f=1$, BM25's TF factor is $2.2/[1+1.2(0.25+7.5)]\approx0.213592233$. It is $1$ at average length and also $1$ at $b=0$ for this frequency. The orthogonal-section cosine factor $0.316227766$ belongs to a different model; the numbers illustrate tunable versus fixed assumptions, not matched-corpus retrieval quality or a universal ordering of penalties.

Practical TF-IDF decisions

• BM25 is a useful sparse baseline, not a theorem that TF-IDF is obsolete. TF-IDF can work well in some pools, especially with a later reranker; that reranker cannot recover candidates excluded by first-stage truncation.
• If using TF-IDF, test sublinear TF, normalization, analyzer and query weighting together. Changing one flag does not guarantee better retrieval.
• Merging fragments below roughly $50$ tokens is a source heuristic; short identifiers, headings, glossary entries and FAQs may be exactly the desired evidence.
• Audit the actual length distribution before tuning. A target of $512$ tokens does not guarantee uniform chunks; source examples describe $5$- to $10$-fold spreads and a $95$th percentile around five times the mean.
• Maintain document frequencies and document norms consistently with the searchable collection and snapshot, especially after major ingest, deletion or analyzer changes.
• Multi-tenant statistics trade estimation stability, relevance, isolation and possible information leakage. Global statistics are not universally preferable to tenant-conditioned ones; choose and validate a policy that respects access boundaries.

18.3 BM25 saturation and length normalization

BM25 separates rarity from within-document evidence

The source identifies BM25 with the Okapi Best Match family. The unboosted, distinct-query-term baseline used here is:

Rarity and within-document evidence are separate factors. Take $F(0,\ell)=0$, $k_1\gt 0$, $b\in[0,1]$ and $\ell\gt 0$. The count $n_t$ is a document frequency, not a term occurrence total. Query boosts, repeated-query-term weights, field mixtures and engine-specific norm encodings require their own formulas and bounds.

$k_1$ gives a limiting repetition budget at average length

At $\ell=1$, $F(f,1)=f(k_1+1)/(f+k_1)$, so $F(1,1)=1$ and $\lim_{f\to\infty}F(f,1)=k_1+1$. For finite counts $f\ge1$, the range is $[1,k_1+1)$, with the upper endpoint excluded. Thus later occurrences share a limiting increment $k_1$. At another length the first occurrence is $(k_1+1)/(1+K_d)$, so the same first-occurrence budget interpretation does not transfer unchanged. A curve that varies $f$ while holding $\ell$ fixed is a mathematical comparison; adding words to one real document can also alter $\ell$.

The default curve spends its budget early

At $k_1=1.2$ and $\ell=1$, $F=2.2f/(f+1.2)$. For $f\in\{1,\,2,\,5,\,10,\,20\}$, the factors are approximately $1$, $1.375$, $1.774193548$, $1.964285714$ and $2.075471698$. The second occurrence adds $0.375$; moving from ten to twenty adds approximately $0.111185984$. Counts five and ten use $(F-1)/1.2\approx64.5161\%$ and $80.3571\%$ of the limiting later-occurrence budget. Those percentages are not shares of the whole term score.

The two-Poisson argument gives the curve meaning

The source cites Robertson and Walker (1994) for the two-Poisson motivation. Do not equate a saturating probability with a saturating log odds: the log likelihood ratio between two individual Poisson distributions is linear in the count. A simple mixture illustrates how a finite relevance-evidence limit can arise. Suppose, at fixed length,

The high-rate component dominates both mixtures at large $f$, producing that finite limit under these particular assumptions. BM25 uses a convenient rational TF shape rather than exactly computing this mixture. The derivation does not make BM25 a calibrated posterior, and changing the mixture assumptions can remove the finite limit.

Log damping loses because it remains purchasable

With natural logs, $g(f)=1+\ln f$ gives approximately $1$, $1.693147$, $2.609438$ and $4.218876$ at $f\in\{1,\,2,\,5,\,25\}$. At $f=10{,}000$, $g(f)\approx10.210340$, about $6.030392$ times $g(2)$. These are unnormalized TF factors. The corresponding BM25 factor at $\ell=1$, $k_1=1.2$ remains below $2.2$. Bounded positive term contributions support safe pruning when the implementation's upper bounds also account for boosts, fields and norm encoding; this is not a statement that normalized TF-IDF cosine is unbounded.

$b$ interpolates length adjustment

Let $\ell_d=L_d/\overline L$. BM25 places the length adjustment inside the TF denominator:

For fixed $f\gt 0$, compare against average length:

For $\ell\gt 1$ this ratio is at least $1/\ell$: the penalty is no stronger than direct division by length. For $0\lt \ell\lt 1$ the ratio is at most $1/\ell$: the short-document bonus is no larger than direct division. At $\ell=1$ both equal $1$. Thus softer means less penalty for long documents and less bonus for short ones, not that BM25 lies below a division curve everywhere. The scorer still cannot identify scope versus verbosity from length alone.

The twice-average example shows the soft discount

At $k_1=1.2$, $b=0.75$, $\ell=2$ and $f=1$, $K=1.2(0.25+1.5)=2.1$ and $F=2.2/3.1\approx0.709677419$. Relative to its average-length value $1$, the discount is approximately $29.0323\%$. Direct division gives $0.5$, a $50\%$ discount. This is a comparison of defined TF factors, not an empirical quality result.

The IDF guard avoids negative rarity weights

Raw zero-feedback RSJ $w_t^{(0)}=\ln((N-n_t+0.5)/(n_t+0.5))$ is negative for $n_t\gt N/2$. The guarded form instead uses $I_{\mathrm B}(t)=\ln(1+\exp(w_t^{(0)}))$, positive for $0\le n_t\le N$. It is a further transformation, not the identical zero-judgment limit. Rare positive weights change little, while common terms change substantially. Although this transform is monotone for a single term, applying it before summing multiple matched terms need not preserve document order.

The release-note example separates the two constants

• The source's motivating vignette puts a $3{,}000$-token runbook at rank $1$ and a $120$-token release note at rank $40$. Those ranks require the other candidates and are not deducible from this two-document calculation.
• Take $N=10^6$, $\overline L=300$ tokens and frequencies $(400{,}000,\,250{,}000,\,5{,}000)$ for kubernetes, pod and eviction. Guarded IDFs are approximately $(0.916290482,\,1.386293361,\,5.298218372)$.
• A has length $120$, $\ell_A=0.4$ and counts $(1,1,2)$. B has length $3{,}000$, $\ell_B=10$ and counts $(12,30,25)$.
• Define this example's separate single-IDF linear baseline $s_{\mathrm{lin}}(d)=\sum_{t\in Q(q)}f_{t,d}I_{\mathrm B}(t)$, equivalent to unit query indicators against these document coordinates. It is not the clinical symmetric TF-IDF score. It gives $s_A\approx12.899021$, $s_B\approx185.039746$, a B-to-A ratio approximately $14.345255$.
• Direct division by relative length gives $s_A/0.4\approx32.247551$ and $s_B/10\approx18.503975$, an A-to-B ratio approximately $1.742736$. This reversal isolates an intentionally different length rule.

BM25 creates a near-tie, not a candidate-pool guarantee

At $k_1=1.2$, $b=0.75$, $K_A=0.66$ and $K_B=9.3$. In query-term order kubernetes, pod, eviction,

Totals use unrounded contributions. B's relative lead is approximately $1.217953\%$ of A's score, not a probability margin. The pair is close compared with the other two scoring rules, but both reach a reranker only if the rest of the ranking and candidate cutoff permit it.

Sweeping $b$ isolates length sensitivity

With $k_1=1.2$ fixed, at $b=0$ the scores are approximately $9.587634$ and $15.887339$ for A and B; at $b=1$ they are approximately $12.822825$ and $11.062111$. The pair ties at $b\approx0.768972775$ under these exact inputs. Increasing $b$ raises A and lowers B because their relative lengths lie on opposite sides of $1$; it does not make that direction correct for every query. A benchmark default is a starting point, not a diagnosis.

Score bounds provide sanity checks and pruning

• For $f\gt 0$, $\ell\gt 0$ and the displayed positive-weight baseline, $F(f,\ell)\lt k_1+1$ and a term contribution is below $(k_1+1)I_{\mathrm B}(t)$. With boosts or field combinations, derive a corresponding bound instead of using this one unchanged.
• Eviction's limiting contribution here is $2.2I_{\mathrm B}(\text{eviction})\approx11.656080$. A contributes approximately $8.763970$, or $75.187970\%$ of that limit; B approximately $8.495685$, or $72.886297\%$.
• These two shares differ in both count and relative length. Their difference is not an isolated causal effect of twenty-three extra occurrences.
• The source describes Lucene versions that omit the common $(k_1+1)$ multiplier. If it is the same positive constant for every scored term/field, dividing these totals by $2.2$ gives approximately $5.370722$ and $5.436134$, preserving rank and relative margin. Different field parameters, fusion scales and absolute thresholds need separate treatment. Check the actual engine version and score explanation.

Practical BM25 decisions

• Start experiments at $k_1=1.2$, $b=0.75$ if appropriate, then use held-out queries and per-term explanations. Tune length effects before assuming repetition caused the failure.
• For $A_\ell=1-b+b\ell$ and $D=f+k_1A_\ell$, the sensitivities are $\partial F/\partial b=-f(k_1+1)k_1(\ell-1)/D^2$ and $\partial F/\partial k_1=f(f-A_\ell)/D^2$. Thus raising $b$ penalizes long documents and rewards short ones at fixed $f$, while changing $k_1$ is not a uniform score multiplier.
• A range such as $k_1\in[1.2,2.0]$ is a search heuristic. Higher counts may indicate depth or stuffing; a development set decides whether more repetition reward helps.
• Length-spread thresholds around threefold or tenfold are workload heuristics, not mathematical inertness boundaries. If all $\ell=1$, $b$ is exactly inert; otherwise sensitivity also depends on frequency and $k_1$.
• Chunk design changes length spread, recall and evidence boundaries. Tune on actual heading, section and table chunks rather than an assumed uniform target.
• Across shards, use compatible searchable-collection statistics and score semantics when a global ranking is intended. Global or distributed frequency collection has latency and multi-tenant isolation trade-offs.
• A one-shard rule below a few million chunks is not universal. The source names dfs_query_then_fetch as one version-dependent distributed-statistics option; measure its extra work and round trips.
• Keep $N$, $n_t$ and $\overline L$ consistent with the index's refresh, deletion and snapshot policy. Some engines maintain statistics incrementally; they do not all wait for a complete re-index, and they need not recompute a full corpus on every write.
• Raw BM25 scores are not calibrated across queries or retrievers. Rank fusion is one robust candidate, not a substitute for evaluating hybrid retrieval and score calibration.

18.4 Probabilistic models and the binary independence assumption

The Probability Ranking Principle starts the derivation

The source cites Robertson (1977) for PRP. Under fixed selection costs and additive expected utility, ranking by marginal relevance probability is optimal for the corresponding objective. For decreasing nonnegative position weights $v_j$, $\mathbb E[U(\pi)]=\sum_j v_j\Pr(R_{\pi_j}=1\mid d_{\pi_j},q)$ is maximized by sorting those probabilities. Linearity of expectation does not require independent relevance events across documents. Odds are a convenient strictly increasing transform for probabilities in $(0,1)$; handle endpoints by limits and ties explicitly.

A RAG context often has non-additive value: a second copy of the same fact may add little even if each chunk is relevant. That breaks the additive objective, not necessarily independence of relevance labels. Diversification, MMR or explicit set selection addresses a different utility. Query-level prior odds in this model are constant across documents and can be dropped for within-query ranking; document-specific priors or features cannot simply be canceled.

Binary independence makes document likelihood estimable

Represent a document by $x\in\{0,1\}^{V}$. Assume its term incidences are independent conditional on $(R,q)$, and let $p_t=\Pr(x_t=1\mid R=1,q)$ and $u_t=\Pr(x_t=1\mid R=0,q)$, with $0\lt p_t,u_t\lt 1$. Then:

Binary incidence discards counts; conditional term independence discards dependencies beyond what the relevance class explains. The model uses per-term parameters instead of a general joint distribution, but assumes rather than proves that they suffice for relevance. Phrase or interaction information requires additional features or a different model.

Off-query terms collapse into a constant

For the unexpanded query-term BIM, additionally assume $p_t=u_t$ for all $t\notin Q(q)$. Expanding the remaining likelihood factors gives:

The last product is document-independent for a fixed query and model, leaving a sum over matched query terms after logarithms. This unexpanded BIM cannot reward a non-query term. That is not a limitation of every sparse method: query expansion, phrase features, relevance models and learned sparse representations can introduce additional coordinates.

The RSJ weight is a log odds ratio

Take natural logs and drop only the document-independent factor:

Each weight compares term-presence odds in relevant and non-relevant documents. A zero or unit probability makes some odds undefined or infinite; smoothed finite estimates avoid that numerical failure, not all estimation bias. One two-by-two table per term gives the following estimator.

The contingency table yields the full estimator

In a fully specified binary-labeled population of $N$ documents, let $S$ be relevant, $n_t$ contain the term and $s_t$ satisfy both. Cells are $a=s_t$, $b_{\mathrm{cell}}=S-s_t$, $c=n_t-s_t$ and $d_{\mathrm{cell}}=N-S-n_t+s_t$. Feasibility requires $\max(0,S+n_t-N)\le s_t\le\min(S,n_t)$. If only some positives are judged and the remaining corpus is unjudged, treating all other documents as non-relevant is an additional proxy assumption, not a verified relevance partition. The worked example makes that proxy explicit. Adding $0.5$ to every cell gives:

Zero feedback gives raw RSJ IDF, not the guarded form

Algebraically set $S=s_t=0$ in this smoothed estimator. The relevant-row pseudocounts are both $0.5$ and cancel:

The half-count choice and the zero-feedback probability model are assumptions, not a theorem of maximal ignorance with no tuning choices. This limit relates a classic IDF to a term-evidence model. Guarded BM25 instead uses $I_{\mathrm B}(t)=\ln(1+\exp(w_t^{(0)}))$, so its exact formula requires that further design choice.

Small relevance sets can increase variance

Feedback can help learn relevance-specific term evidence, but small positive samples and small cells produce unstable estimates. Pseudo-relevance feedback can amplify selection errors. In the toy example, changing the ferry incidence count by one among $S=20$ positive examples moves its weight by approximately $1.148649$ nats. Keeping $S$ fixed is not equivalent to flipping one binary relevance label; that distinction matters in the arithmetic.

BM25 adds count and length evidence without explicit term interactions

BM25's rational TF factor and length normalization extend a binary-incidence baseline; they do not certify that every missing assumption has been repaired. An unexpanded unigram score does not explicitly model phrase order or term co-occurrence. A full pairwise vocabulary model has $V(V-1)/2=O(V^2)$ possible pairs, while a spanning tree itself has only $V-1$ edges. Building a Chow–Liu-style tree may estimate all pairwise statistics, but it is incorrect to call the tree an $O(V^2)$-edge dependency structure. Late interaction and cross-encoders offer other trade-offs.

Binary representation is not binary relevance

Two different binaries occur here: the relevance variable $R\in\{0,1\}$ and term incidence $x_t\in\{0,1\}$. Adding graded TF does not add graded relevance labels. If relevance is graded, for example $0$ through $4$, one can rank by expected gain under a chosen utility, requiring a correspondingly extended probability or learning objective. Graded labels therefore can change a probabilistic derivation as well as a learning-to-rank target; they do not automatically follow from replacing binary incidence with TF.

The Irish Sea example shows feedback inversion

Take $N=10^6$ and query irish sea ferry with frequencies $(8{,}000,\,120{,}000,\,40{,}000)$. At $S=0$, raw RSJ weights are approximately $(4.820219572,\,1.992426566,\,3.178041851)$ nats. A containing irish and sea scores $6.812646138$; B containing sea and ferry scores $5.170468417$. A leads by approximately $1.642177720$ nats. These are binary RSVs, not the guarded BM25 TF scores.

Twenty judgments reverse the order

For a feedback scenario, use $S=20$ with $(s_{\mathrm{irish}},s_{\mathrm{sea}},s_{\mathrm{ferry}})=(8,14,20)$, keeping the corpus frequencies fixed and treating the rest of the corpus as the non-relevant proxy. The full estimator gives weights approximately $(4.435545432,\,2.794882894,\,6.892114037)$. Ferry's odds ratio is approximately $984.480446718$, so write $w_{\mathrm{ferry}}=\ln(984.480446718\ldots)$, not an exact equality to the log of a rounded input. A scores $7.230428325$ and B $9.686996930$, a B lead approximately $2.456568605$ nats. This illustrates one feedback inversion, not evidence that this small sample generalizes.

One incidence-count change exposes sensitivity

At fixed $S=20$, change $s_{\mathrm{ferry}}$ from $20$ to $19$, leaving the other counts fixed. This can represent swapping one positive example for another that differs in ferry incidence, not flipping just one document's relevance label while somehow keeping $S$ unchanged. The ferry weight becomes approximately $5.743465274$. Its shift is approximately $1.148648763$ nats, and B's lead becomes $5.743465274\ldots-4.435545432\ldots\approx1.307919842$ nats. The shift is approximately $40.619973\%$ of the initial max-to-min query-weight spread $2.827793005$ nats. A real label flip must also update $S$ and any affected counts.