20. Dense Retrieval

00-START-HERE · 01-READING-ROUTES · 19_learning_to_rank · 21_learned_sparse_and_multi_vector_retrieval This chapter prepares a Retrieval-Augmented Generation (RAG) candidate to explain dense retrieval, train it, price it, combine it with lexical search, and extend it across languages.

TL;DR

• A cross-encoder conditions passages on the current query; a bi-encoder precomputes passage vectors. For the stated corpus, the ideal cross-encoding proxy is $36.26$ minutes versus a flat payload-read proxy $32.28\,\mathrm{ms}$, not matched measured latencies.
• At $B=128$, a two-passage-per-example pool gives $255$ negatives before masking. Reusing all $256$ passage encodings saves $128$ times passage-encoding work compared with independently encoding the same-size pool for every question; this is not an end-to-end speedup.
• Exact lexical matches and learned semantic matches have complementary possible failures. Opposite IDF and toy exposure trends do not establish a document-frequency performance crossover or near-disjoint error sets.
• A margin-one negative has $e^7\approx1{,}096.63$ times a margin-eight negative's unnormalized score weight. Parameter-gradient norms also depend on score Jacobians; ANN mining is a heuristic, not the exact variance-optimal distribution.
• Document crops supply proxy positives without human labels. A MoCo queue stores $65{,}536$ fp32 width-$768$ keys in $201.326592\,\mathrm{MB}$, but stop-gradient and stale keys make it different from a fully differentiable current-encoder batch.
• Multilingual vectors can mix language identity and semantics. Per-language centering and language-controlled negatives can help under the stated model; they need native-query and true cross-lingual evaluation.
• Compare dense/sparse candidate unions and shallow reranking on final quality and measured serving tails. Precomputation, payload sizes and nominal translation times do not alone establish a production SLO.

The story

Imagine a parcel hub that routes millions of parcels when a customer asks a question. A cross-encoder clerk opens the request and parcel together, allowing detailed query-dependent inspection. A bi-encoder crew instead prepares each parcel's vector card before requests arrive and compares that card with a query vector. The card is reusable because it does not depend on the current question; the price is a fixed representation, not a universal claim about model capacity or accuracy. The hub preserves an exact-string lane for codes and quoted material, while a learned lane can connect paraphrases. Which lane succeeds depends on training, analysis, data and the query—not a guaranteed frequency boundary. Training contrasts a chosen positive with negatives. A near miss can receive a much larger softmax weight than an easy example, but the actual parameter update also depends on how both scores change with the parameters. ANCE retrieves the model's near misses from a checkpointed index. Refreshing that index limits staleness at a real offline compute cost; highly ranked unlabeled positives must not automatically be treated as wrong parcels. Without labeled requests, the crew can create two cropped views of one document as proxy positives. This assumes those views share useful information; they may overlap, and even disjoint spans may repeat words. A MoCo key encoder supplies a FIFO queue without retaining old encoder activation graphs. The queue saves negative-vector storage relative to fully retained activations, but its stopped gradients and temporal drift differ from a fresh large batch. Negative count affects a contrastive information-bound certificate only under its sampling assumptions; it does not cap everything the model can learn. In a multilingual hub, a vector may also carry a language direction. Under an explicit toy model, that direction can make a same-language distractor outrank a cross-language answer. Removing estimated means and controlling negative languages are hypotheses to evaluate, not guaranteed fixes on arbitrary corpora. Translation placement adds another choice: offline translation volume affects price, while online autoregressive output affects latency. The final design keeps provenance and access checks, retrieves a measured candidate union, and spends joint scoring only where it demonstrably helps.

Decoder table

Core mechanics

Mathematical conventions

Let $N$ be corpus passage count, $P$ parameters per encoder, $d$ vector width, and $L_q,L_p$ tokenized input lengths. Separate towers may share weights or have $2P$ total parameters. The proxies $W_{\mathrm{forward}}\approx2PT$ and $W_{\mathrm{train}}\approx6PT$ for $T$ processed tokens omit architecture- and sequence-length-dependent work. A multiply-accumulate counts as two FLOPs. Dividing by stipulated throughput $R$ or bandwidth $\mathcal B$ gives conditional work proxies, not measured latency, throughput, invoices or a $p_{95}$ guarantee. Corpus encoding, index construction, graph links, text, identifiers, replicas and runtime buffers are different costs. Use decimal units: $1\,\mathrm{MB}=10^6\,\mathrm B$, $1\,\mathrm{GB}=10^9\,\mathrm B$, $1\,\mathrm{TB}=10^{12}\,\mathrm B$. fp32 is $4\,\mathrm B$ and fp16 $2\,\mathrm B$ per value. Retain full precision until reporting; $\approx$ marks rounding or a model approximation. Logarithms are natural, in nats, unless explicitly $\log_2$ for bits. Logits are dimensionless; a raw dot-product gap is not information measured in nats merely because it appears in an exponential. For gold passage set $G_q$ and returned set $S_{q,k}$, $\mathop{\mathrm{Recall}}\nolimits @k=\lvert G_q\cap S_{q,k}\rvert/\lvert G_q\rvert$ when $\lvert G_q\rvert\gt 0$. Answer-hit accuracy instead averages $\mathbf1[G_q\cap S_{q,k}\ne\varnothing]$ over questions, using the declared answer-matching rule. DPR's cited top-$k$ accuracy is this question-level measure, not automatically passage recall. ANN neighbor recall compares approximate results with exact top neighbors and is a third metric. A reranker cannot return an excluded item; its gain-dependent NDCG ceiling is not numerically its input's recall. For the contrastive loss, let $M$ be the number of eligible negatives, $\Delta_j=s^+-s_j^-$ the score margin, and $Z=1+\sum_j e^{-\Delta_j}$. If a temperature is used, margins in this expression are divided by $\tau\gt 0$ and derivatives gain $1/\tau$. Positive score-softmax weights do not by themselves specify parameter-gradient norms. Importance-sampling arguments below condition on a fixed full objective and its normalizer; sampled-softmax or truncated mining generally changes that objective. For in-batch reuse, $B$ counts query-positive examples; a two-passage-per-example pool has $2B-1$ eligible negatives per query only before duplicate and false-negative masking. For MoCo, $K$ counts queued negatives, so there are $K+1$ logits. The queue-turnover/EMA comparison is a design heuristic, not a hard stability theorem. Match sequence lengths, trainable branches and processed anchors when comparing costs. The language-centroid toy model assumes unit centroid directions, orthogonal unit semantic residuals, a common amplitude $0\le\alpha\lt 1$, and zero within-language residual mean. Language identity occupies squared norm share $\alpha^2$, not $\alpha$. Empirical means, topic mixtures, language distributions and optional renormalization can violate this toy geometry; centering needs validation rather than an unconditional production prescription. Reported benchmarks, prices, hardware rates and historical claims are not independently verified here.

20.1 Bi-encoders, DPR, and the efficiency-expressiveness trade

Joint scoring versus independent scoring

What: A cross-encoder scores the concatenated query and passage.

What: A bi-encoder scores two independently computed vectors.

Why: $E_P(p)$ is independent of the current query and may be computed at index time. Its version includes weights, tokenizer, chunking, pooling and normalization, not just a model name. Query-independent document work can also exist in late-interaction architectures; the single-vector formula describes one particular factorization. Failure at corpus depth: A standard joint cross-encoder must condition on the query for each candidate; with $N$ candidates, its pair-work count is linear in $N$. Batching changes utilization and call count, not the number of scored query–passage examples in this baseline. Trade-off: Single-vector separation removes online token-level query conditioning of passage states. It does not prove equal parameter capacity: two untied $P$-parameter towers have $2P$ parameters, while the joint example has one $P$-parameter encoder. Comparisons require an explicit capacity and compute budget. A width-$768$ passage vector must support future questions through the chosen vector score. A joint encoder can form a $32\times128=4{,}096$ query-to-passage attention-score grid per head and layer; the opposite direction and within-query/within-passage attention are additional. This grid is not the total attention FLOP count, and special tokens are omitted from this length proxy.

Corpus-depth arithmetic

Use the approximate forward rule $2P$ FLOPs per processed token, with $P=110\times10^6$ parameters per BERT-base encoder, $L_q=32$ and $L_p=128$. These token lengths are stipulated independently of the prose description of a hundred-word passage. Joint attention, embedding/head details and serving overhead are omitted:

With the full corpus count $N=21{,}015{,}324$, not a rounded $21$ million inside the calculation, the two online arithmetic proxies are:

Why: The arithmetic ratio is approximately $1.881353263\times10^7$, about seven orders of magnitude. It excludes the bi-encoder's offline passage encoding, stored index, memory movement, top-$k$ selection and index traversal; it is not the query-latency ratio. Cost proxy: At $R=3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$, $W_{\mathrm{cross}}/R\approx2{,}175.704132\,\mathrm s\approx36.261736\,\mathrm{min}$ per query. Claim limit: The throughput and $2P$ rule are assumptions, not a benchmark or a rigorous timing lower bound. Long-sequence attention, memory, execution efficiency, batching and I/O change the real result.

DPR loss and in-batch negatives

What: For query $q_i$, a chosen positive $p_i^+$ and $M_i$ eligible negatives $p_{i,j}^-$, DPR uses a contrastive softmax over dot products. The following equation uses temperature one; a different temperature must rescale every logit consistently.

Why: The training objective encourages separately computed vectors to meet the supervision task. It does not recreate arbitrary query-conditioned token interaction at inference or guarantee relevance generalization; augmentation, labels, negatives and architecture still matter. Sampling: Reuse passage encodings across all questions in a batch. Other examples' positives and hard negatives are candidates for negatives, not verified negatives for every query; mask known shared positives, duplicates and invalid candidates. At $B=128$ with one positive and one hard-negative passage per example, the shared pool has $2B=256$ encoded passages. Each question has one chosen positive and $2B-1=255$ remaining negatives before masking. Independently encoding a same-size positive-plus-negative pool for each question costs $128\times256=32{,}768$ passage encodings instead of $256$, exactly $128$ times the passage-encoding work. Counting only $128\times255=32{,}640$ negatives on one side but all $256$ passages on the other gives $127.5$, an unmatched comparison. Query encodings and pair-score arithmetic remain; including the stipulated query/passage token work changes the encoder-work ratio to approximately $113.888889$, before other costs. Neither ratio counts batched network calls or guarantees a wall-time speedup. Failure without useful negatives: The training objective becomes easy while live-corpus recall stays poor.

Retrieval is not symmetric sentence similarity

Task distinction: The source describes Sentence-BERT training using SNLI classification and STS cosine regression. STS similarity is symmetric, but natural-language entailment itself is directional; do not describe every component as symmetric supervision. A model name or shared encoder does not uniquely determine its retrieval objective. Why transfer can fail: A short question and a longer passage need not be paraphrases to be relevant. Retrieval supervision must teach that relationship rather than merely sentence likeness. A shared encoder with different role prompts or task training can still serve retrieval; untied towers alone are not a quality guarantee. Comparison: The cited DPR recipe trains query and passage encoders on question–passage relevance. Evaluate the objective and data, not just whether tower weights are tied. The source reports Natural Questions top-$20$ answer-hit accuracy $78.4\%$ for DPR versus $59.1\%$ for BM25, a $19.3$-percentage-point gap in the cited setup. These are question-level answer hits, not passage-level recall against an exhaustive gold set. Claim limit: This comparison applies to the cited Natural Questions setup and does not establish universal dominance.

Worked deployment configurations

The worked deployment uses $N=21{,}015{,}324$, $d=768$, fp32 vector payload, the token lengths and per-encoder parameter count above, and a nominal $50\,\mathrm{ms}$ target. The following architectures are compared through work proxies, not matched measured latencies. Configuration 1: Full joint scoring has the $7.397394048\times10^{17}$-FLOP proxy and the $2{,}175.704132\,\mathrm s$ arithmetic division. That division is approximately $43{,}514.082635$ times the $0.05\,\mathrm s$ target. Batching may improve utilization; it does not remove corpus-linear pair work in this baseline or turn the proxy into a latency measurement. Configuration 2: A flat bi-encoder index occupies:

Dot products cost $2Nd=32{,}279{,}537{,}664$ FLOPs, whose arithmetic division is $94.939817\,\mathrm{\mu s}$. A complete uncached payload pass at $\mathcal B=2.0\times10^{12}\,\mathrm{B\,s^{-1}}$ takes $32.279538\,\mathrm{ms}$ by division. The read/compute quotient ratio is exactly $340$ under these constants. This suggests a bandwidth bottleneck for this flat resident scan, not a measured universal limit. Halving $d$ halves these two work terms; it does not necessarily halve end-to-end latency. Doubling arithmetic throughput leaves the payload-read division unchanged, not every component of serving. Capacity, data placement, query encoding, selection, kernels and overhead require separate accounting. Configuration 3: An HNSW candidate stage may visit far fewer vectors, then a cross-encoder reranks $k=100$. The visited count and quality depend on data, graph parameters, filters and search settings; “a few thousand” is not an ANN guarantee. The hundred-pair work proxy is $3.52\times10^{12}$ FLOPs, with arithmetic division $10.352941\,\mathrm{ms}$. Stipulating an additional $5\,\mathrm{ms}$ inclusive query/ANN stage gives a nominal $15.352941\,\mathrm{ms}$ budget sum, not a measured total. Its cross-encoder pair count is $N/100=210{,}153.24$ times smaller than full-corpus joint scoring. ANN recall, queueing, batching, transfer and reranker utilization must be measured. Reported throughput check: The source cites DPR at $995$ questions per second over a roughly twenty-one-million-vector FAISS index, and BM25/Lucene at $23.7$ questions per second. Those are throughput observations under their stated experimental conditions. The numerical throughput ratio is approximately $41.983122$. Reciprocal throughput $1/995\approx1.005025\,\mathrm{ms}$ is an average completion-spacing or amortized work figure, not necessarily single-request latency when queries overlap or batch. The query arithmetic proxy $W_q/R\approx20.705882\,\mathrm{\mu s}$ is numerically $2.060235\%$ of that reciprocal, but comes from an assumed accelerator rate, not a matched component profile. This does not establish that the index rather than encoding or other overhead dominates the published system.

Practical boundaries

A retrieval–reranking funnel is a useful baseline. Under only the pair-FLOP proxy, a $50\,\mathrm{ms}$ arithmetic allowance buys $\lfloor0.05R/W_{\mathrm{pair}}\rfloor=482$ pairs; five hundred pairs already give $51.764706\,\mathrm{ms}$ before omitted work. Thus “roughly five hundred documents” is a planning illustration, not a guarantee that a small corpus meets the target. Benchmark full joint scoring if the collection is small enough. Fine-tune with suitable in-domain pairs and compare tied, untied and role-prompted encoders. Fewer than roughly a thousand labels is a source heuristic for trying an unsupervised initializer, not a theorem that fine-tuning loses. Increasing $B$ from eight to $128$ changes the unmasked $2B-1$ pool from $15$ to $255$; quality does not depend on count alone. Symmetric tasks can use shared encoders, while retrieval still needs task-matched supervision. Price nominal vector payload as $Ndb$ bytes per coordinate width $b$, with index/runtime overhead separate. The fp16 payload here is $32{,}279{,}537{,}664\,\mathrm B=32.279537664\,\mathrm{GB}$. Report task passage recall or answer-hit accuracy with its actual definition, ANN neighbor recall separately, and a final ranking/answer metric. An answer-bearing passage at rank $87$ survives a hundred-candidate cut; survival does not certify the final answer.

Re-indexing when the model changes

When the passage embedding function changes incompatibly, existing passage vectors no longer belong to the new scoring contract. Query-only updates can sometimes reuse a fixed passage index, and an explicitly valid vector-space transformation is another special case; validate compatibility rather than assuming every parameter edit forces re-encoding. For a full rebuild of $500\times10^6$ chunks at $256$ tokens, the encoding work proxy is:

Division by $R$ gives $82{,}823.529412\,\mathrm s\approx23.006536\,\mathrm h$ on one stipulated accelerator, or $1.917211\,\mathrm h$ on twelve only under perfect scaling. The width-$768$ fp32 payload is $1.536\,\mathrm{TB}$; keeping old and new payloads gives $3.072\,\mathrm{TB}$. Graphs, source data, replicas, build buffers and dual-write ingestion are additional. Version the tokenizer, chunking, encoder, pooling, normalization, any language centroids and search metric, then use a validated cutover. Decision boundary: A shadow change smaller than a well-specified uncertainty interval has not demonstrated a benefit, but is not proof of equivalence. Use paired query analysis, important cohorts, serving compatibility and rebuild cost rather than an unspecified “noise band.”

20.2 Where dense wins and where BM25 wins

Opposite dependence on document frequency

What: A standard lexical BM25 sum includes terms present in the analyzed query and document. This chapter uses the positive guarded IDF convention from Chapter 18: Its inverse document frequency is:

Why: With fixed $N$ and $0\le n(t)\le N$, this guarded weight is positive, finite and decreases with frequency. Raw RSJ log odds and log odds plus one are other conventions, not algebraically the same formula. The examples below consistently use the guarded version; benchmark results are not recomputed from these toy weights. Failure without overlap: If every analyzed query term has zero frequency in a candidate, this lexical sum is zero, absent additional signals or boosts. Synonym expansion or a different analyzer can change the overlap; tuning term saturation alone cannot create it. On the $21{,}015{,}324$-passage corpus, $n(t) = 3$ gives:

At $n(t)=100{,}000$, $\mathop{\mathrm{IDF}}\nolimits_+(t)\approx5.347832026$, so the rare/frequent IDF ratio is approximately $2.918565775$. These are dimensionless scoring weights formed with natural logarithms, not measured mutual information or full-document BM25 ratios; term frequency and length factors also matter.

Dense supervised exposure

Toy exposure model: Suppose each positive training passage is sampled independently and uniformly from the corpus. Then training count $T$ gives expected term sightings $Tn(t)/N$. Real query-positive supervision is not uniform; pretraining, subwords, related terms and generalization also affect dense behavior. For $T=58{,}880$ training pairs and $n(t)=3$, that explicit toy model gives:

Under the same independent-draw assumption, the probability of at least one sighting is $1-(1-3/N)^{58{,}880}\approx0.008370070$, about one in $119.473314$. An expectation alone is not that probability. With $n(t)=100{,}000$, expected count is approximately $280.176504$; this is not a measured count of useful supervised examples. Why it can help: Repeated useful patterns can receive more training support, but the count above does not prove a dense accuracy curve or a universal relation between frequency and competence. Possible failure: New identifiers, version strings and names can remain poorly supported or difficult to preserve exactly. Subword representation makes them encodable, not guaranteed to be retrieved correctly. Comparison: Guarded IDF decreases with frequency, while the toy expected sighting count increases. Their units and functions differ; neither one is a retrieval-accuracy curve. Equating these two arbitrary numeric scales would give $n\approx3{,}143.552230$ for the guarded formula and these constants. That numerical intersection is not a dense-versus-BM25 performance crossover. It moves under score conventions or rescaling without changing model competence. Only frequency-stratified evaluation can locate an empirical preference region; no exact boundary or near-disjoint error sets follow from these equations.

Paraphrase and rare-string regimes

A semantic encoder may connect a waterway paraphrase to an Irish Sea passage even without exact query wording. A lexical retriever may fail when the analyzed terms do not overlap. Conversely, an exact rare code can receive high lexical weight while several distinct codes occupy nearby dense positions. Neither paraphrase success nor rare-string success is guaranteed by architecture, and high IDF alone does not prove a document outranks every competitor. Bigger-model limit: Increasing width changes capacity and geometry but does not create new observed examples by itself. It can still improve retrieval; absence of occurrences is not a theorem that extra width cannot help through generalization. Thakur et al. report BM25 as a strong zero-shot baseline that most dense retrievers failed to beat across $18$ BEIR datasets out of domain. Claim limit: The chapter uses this as a transfer result, not a claim that BM25 wins every dataset.

Enterprise support worked example

The constructed support example has $N=2\times10^6$ documents and $d=768$. Assume the SSO-certificate query and the relevant SAML/signing-key passage share no terms after the specified analysis; the lexical sum is then zero. Its dense rank one is an assumed outcome, not a deduction from a paraphrase. For ERR_SSL_5417 remediation, assume the analyzer preserves the code as a searchable literal with document frequency four:

For remediation at frequency $200{,}000$, guarded IDF is approximately $2.302583093$. The code's share of these two IDF weights is approximately $13.004580842/(13.004580842+2.302583093)=84.957481\%$, not necessarily its share of a document's BM25 score once frequency and length factors apply. If all four code-matching passages are relevant and the constructed BM25 top twenty includes all four, passage recall is $100\%$ and answer-hit rate is also $100\%$. That ranking is stipulated, not proved by IDF alone. For a clean comparison, suppose a dense neighborhood has $1{,}800$ candidate passages including those four, and chooses twenty uniformly without replacement. Expected passage recall is $20/1{,}800\approx1.111111\%$, but probability of at least one relevant result is $1-\binom{1796}{20}/\binom{1800}{20}\approx4.374504\%$. Counting distinct codes instead of passages would be a different sampling model. Claim limit: These are constructed candidate and random-selection assumptions, not empirical dense/BM25 guarantees. Distinguish expected recall, probability of a hit and a particular realized ranking.

Hybrid union and benchmark reversal

A union of dense top twenty and lexical top twenty contains at most forty distinct candidates after deduplication. Its maximal forty-pair proxy is $1.408\times10^{12}$ FLOPs, or $4.141176\,\mathrm{ms}$ of arithmetic division; smaller overlap-adjusted unions cost less pair work. The source reports top-twenty answer-hit accuracy $78.4\%$ versus $59.1\%$ on Natural Questions, but $63.2\%$ versus $68.8\%$ on SQuAD, reversing the gap by $5.6$ percentage points in favor of BM25 there. The authors' proposed explanation concerns passage-visible question writing and lexical overlap; these results do not identify a universal winner or prove a causal effect in a different collection. Decision boundary: Collection process can make a benchmark recommend the wrong retriever for a different query process.

Practical boundaries

Compare dense, lexical and hybrid baselines, particularly identifier, rare-term, quoted-text and important tail cohorts. Document-frequency bins are useful slices; a non-dictionary-token detector is only a proxy when corpus statistics are unavailable. A hybrid union may help before increasing width from $768$ to $1{,}536$, but depth choices such as sparse twenty and dense fifty to a hundred are workload heuristics. Sparse hits are not automatically relevant or “near-certain,” so do not always cut dense depth first under a fixed budget. Measure each arm's marginal coverage and final-quality contribution. Track lexical zero-overlap queries and semantic-only hits without equating either with correctness. For a literal-containment requirement, use an exact verified field/string check with a declared normalization and tokenization policy; ordinary BM25 ranking or an analyzer that splits codes does not itself guarantee verbatim containment. Enforce required filters and access constraints before allowing a result into the serving set.

20.3 Hard negatives, gradient norm, and ANCE

Margin form of the contrastive loss

For temperature-one logits, let $s^+$ be the chosen positive score, $s_j^-$ the $j$th negative score, and $\Delta_j=s^+-s_j^-$. The same contrastive loss can be written in margin form:

For $Z=1+\sum_j e^{-\Delta_j}$, define the negative softmax probability $w_j=e^{-\Delta_j}/Z$. Score derivatives and the complete parameter gradient are:

What: A negative's unnormalized score weight is $e^{-\Delta_j}$. Its parameter-component norm is $\lVert g_j\rVert=w_j\lVert\nabla_\theta(s_j^- -s^+)\rVert$, so the margin alone is insufficient. It is not generally the gradient of an independent loss $\mathcal L_j$, because the normalizer couples all scores. If every score only shares one additive-shift parameter, the entire parameter gradient is zero despite nonzero score weights. Why hardness helps: Within one fixed denominator and comparable score Jacobians, smaller margins receive exponentially larger weights. Different denominators, temperatures, Jacobian norms or cancelling directions can change the actual update comparison. For margins eight and one, unnormalized weights are $e^{-8}\approx0.000335462628$ and $e^{-1}\approx0.367879441$. Their ratio is $e^7\approx1{,}096.633158$. Rounding it to $1{,}097$ is reasonable, but it is not an unconditional ratio of parameter-gradient norms or empirical learning efficiency. Failure without hard negatives: A near-zero loss can report an easy task rather than a good retriever.

Variance-optimal sampling

Condition on parameters and a fixed complete negative pool, so $g=\nabla f(\theta)=\sum_jg_j$ is the exact gradient of that objective. Draw one index $J$ with probabilities $p_j\ge0$, $\sum_jp_j=1$, requiring $p_j\gt 0$ whenever $g_j\ne0$. The importance estimator $\widehat g=g_J/p_J$ is unbiased. For an $L_f$-smooth objective and step $\theta^\prime=\theta-\eta\widehat g$, the conditional descent bound is:

For that particular estimator, sampling controls the second moment. Cauchy–Schwarz yields:

When at least one component is nonzero, equality holds at the distribution below, with zero probability allowed for zero components. If all components are zero, any supported sampling rule has zero variance:

Even with this result, an ANN-ranked top list is not the exact optimal sampling distribution. Norms include score Jacobians, while ANN typically ranks scores. Top-$k$ truncation drops nonzero-gradient support, and a list does not specify probabilities or importance correction. Moreover, evaluating $g_j$ for the fixed contrastive objective presumes its complete normalizer; replacing that denominator with a sampled one is generally not an unbiased full-softmax gradient. High-score mining is a useful heuristic motivated by informative examples, not an equality proved by the variance bound. Claim limit: The bound is for a fixed smooth objective with supported importance correction. Practical ANCE or sampled-softmax training can intentionally optimize a different sampled objective. Good retrieval quality is an empirical result, not a consequence of variance-optimality alone.

Dynamic mining with ANCE

What: ANCE mines candidate near misses from an ANN index built using a recent checkpoint. Record which checkpoint encoded the passages and which query encoder retrieves them; approximate search, filters and sampling influence the actual negatives. Why: The hardest useful negative depends on current parameters $\theta _t$. Possible static-mining failure: In an illustrative training trajectory, a negative's margin grows from one to four. This is not an inevitable drift or a universal refresh clock. That change divides its unnormalized coefficient by $e^3\approx20.085537$. A margin-four coefficient is $e^4\approx54.598150$ times a margin-eight coefficient and $e^3\approx20.085537$ below a margin-one coefficient. Normalized weights also depend on the changing denominator, and parameter updates depend on Jacobians. Matching one margin-one coefficient needs roughly $e^7\approx1{,}096.633158$ margin-eight terms in this toy sum; it does not prove that a batch of $128$ must become $140{,}416$, or that summed gradients or training quality would match. Keep query batch size, encoded passage count and negative count distinct. BM25 bootstrap negatives and later self-mining are possible schedules. Switch based on judged hard-negative quality, false-negative rates and held-out retrieval—not a universal requirement that the model first beat BM25 overall. Diversify language, domain and difficulty when hardest-only mining would amplify noise.

False negatives and RocketQA

High-ranked unlabeled passages may be true answers. The source describes RocketQA filtering mined candidates through a cross-encoder to reduce this problem. A teacher can also be wrong; validate filtering thresholds, mask known positives and duplicates, and avoid treating one-positive judgments as exhaustive relevance. Hardness is not the same as a verified negative label.

MS MARCO worked example

For the numerical example, use $N=8{,}841{,}823$, $P=110\times10^6$, $d=768$, $B=128$ and a baseline pool providing $255$ negatives per query. Compare this baseline with one additional negative for each query. The added slot is distinct from replacing an existing slot and may require additional passage encoding. Configuration 1: Suppose all $255$ baseline negative margins are eight. Their total unnormalized weight is $255e^{-8}\approx0.085542970$. Then $Z\approx1.085542970$, loss $\ln Z\approx0.082080295$ nats, and summed negative softmax probability $\sum_jw_j\approx0.078802012$, or $7.880201\%$. This is not the fraction of a parameter-update vector attributable to negatives. The positive score derivative has the opposite total magnitude; parameter Jacobians and cancellation remain relevant. Configuration 2: Add one margin-four negative, weight $e^{-4}\approx0.018315639$. Now $Z\approx1.103858609$, loss $\approx0.098811868$ nats and summed negative softmax probability $\approx9.408688\%$. That score-probability sum is approximately $1.193965$ times Configuration 1, not necessarily that ratio of gradient norms or learning progress. Configuration 3: Instead add one margin-one negative, weight $e^{-1}\approx0.367879441$. There are now $256$ negatives, not $255$. Here $Z\approx1.453422411$, loss $\approx0.373921059$ nats and summed negative probability $\approx31.196878\%$, a ratio $3.958894$ versus Configuration 1. The hard candidate has softmax probability $25.311254\%$ and each easy candidate approximately $0.023080876\%$; neither is a “share of the whole parameter step.” If all component-Jacobian norms were equal, the gradient-norm sampling proxy would allocate the hard negative $e^{-1}/(255e^{-8}+e^{-1})\approx81.133934\%$ of negative draws. Uniform allocation is $1/256=0.390625\%$, a ratio approximately $207.702872$. This allocation is conditional on the equal-norm assumption and full-objective estimator, not ANCE's proved optimality.

Refresh cost and cadence

Encoding a $128$-token passage has proxy $2PL_p=2.816\times10^{10}$ FLOPs. The MS MARCO refresh is $2.4898573568\times10^{17}$ FLOPs, or $732.310987\,\mathrm s$ by division at $R$. This excludes ANN construction, reading/tokenizing the corpus and writing vectors. For the baseline $B$ queries and $2B$ passages, training work is:

The baseline step division is $0.071559529\,\mathrm s$, so one refresh has the same encoder-work proxy as $10{,}233.591435$ baseline steps. At an interval of ten thousand steps, refresh adds approximately $102.335914\%$ to baseline training work, for a total factor $2.023359$. If each query gets one distinct extra mined passage used only in its own denominator, the additive-slot example instead encodes $3B$ passages: $W_1\approx6P(BL_q+3BL_p)=3.514368\times10^{13}$ FLOPs, or $0.103363765\,\mathrm s$ by division. A refresh then equals approximately $7{,}084.794071$ augmented steps. Replacing a passage already in the $2B$ pool avoids this extra encoding but does not add a $256$th negative to the original denominator. Specify the design before comparing cadence. For Wikipedia at the same $128$-token length, re-encoding has proxy $5.9179152384\times10^{17}$ FLOPs, or $29.009388$ minutes on the one stipulated accelerator. The source reports an $8.8$-hour run on eight GPUs: $70.4$ GPU-hours. Its wall time divided by the one-accelerator proxy is approximately $18.201004$, whereas total GPU-hours divided by the proxy GPU-hours is approximately $145.608033$. These are unmatched hardware and system conditions; neither ratio isolates overhead or validates a “FLOP floor.” Claim limit: FLOP divisions are model-based work proxies, not rigorous time bounds or matched published-hardware predictions. Asynchronous workers still consume resources and may contend with training or serving unless isolated. Use asynchronous refresh when its resource budget justifies it. Monitor current-versus-indexed margins, embedding drift, judged negative quality and false positives. A three-logit-unit margin shift is a source heuristic, not a universal time-to-refresh. To keep refresh work at most half of baseline training work, interval $S$ must satisfy $S\ge\lceil2W_{\mathrm{refresh}}/W_0\rceil=20{,}468$ steps under these constants. Twenty thousand gives slightly more than half, not a strict bound. With the augmented-slot $W_1$, the corresponding minimum is $14{,}170$. The mining refresh does not add an online stage by itself, but it is not free aggregate compute or a guarantee of unchanged serving behavior.

Large-corpus boundary

At $500\times10^6$ chunks of $128$ tokens, encoding work is $1.408\times10^{19}$ FLOPs, or $11.503268$ hours by division. This uses half the token length of the earlier $256$-token rebuilding example. A pool of $32{,}768$ margin-eight negatives has the same unnormalized total coefficient as approximately $32{,}768/e^7=29.880548$ margin-one terms, not equivalent parameter directions or training quality. It also processes a different negative pool and activation workload. Uniformly sampling ten percent of the corpus gives an encoding proxy $1.408\times10^{18}$ FLOPs, or $1.150327$ hours, before index work. For a fixed exact global score order and a uniform sample of $m_s$ items without replacement, the expected global rank of its $k$th best item is $\mathbb E[R_k]=k(N+1)/(m_s+1)\approx k/0.1=10k$. This is an expectation, not a rank guarantee or an ANN-recall bound. For a large population and small sampled rank, the rough standard deviation is $\sqrt{90k}$; sample quality varies. Nonuniform mining, ties, false-negative exclusions and approximate search change that model. Treat corpus subsampling as an evaluated compromise, not a reported result.

20.4 Unsupervised dense retrieval: Contriever and MoCo

What labels supplied

DPR supervision supplies a chosen query-positive passage pair; sampled negatives are not exhaustive human judgments. Contriever instead forms two independently cropped views of one document as proxy positives. They may overlap or repeat words; deletion and masking can reduce simple lexical copying but do not guarantee a lexical retrieval floor or remove every shortcut. Crops can also cover different topics, so view construction is a relevance assumption to evaluate. Transfer boundary: The source reports that the evaluated transferred supervised dense retrievers lost to BM25 on most out-of-domain BEIR collections. That is a result for particular models, training data and metrics, not a theorem about every dense encoder or every domain shift. MS MARCO is reported to have $502{,}939$ training queries and $8{,}841{,}823$ passages, a passage/query ratio approximately $17.580309$. This is not the fraction of passages with a unique annotation: one query can have several relevant passages and passages can recur across queries. Matching only this count ratio on an eight-million-document collection would mean approximately $455{,}055$ queries; it does not establish how many labels that product actually needs.

InfoNCE: an information-bound certificate, not a capacity ceiling

Let $X,Y$ be paired augmented views, with one positive from their joint distribution and $K$ negatives independently drawn from the appropriate marginal. Under the standard population InfoNCE sampling and critic assumptions, with expected loss $\mathcal L_{\mathrm{NCE}}$ in nats:

What: Because the contrastive loss is nonnegative, the right-hand certificate cannot exceed $\ln(K+1)$ nats, or $\log_2(K+1)$ bits. This limits that certificate, not the actual mutual information or the encoder's capacity. Correlated, stale, mined or filtered negatives require checking the derivation rather than automatically importing the same IID bound. Training levers: Negative count is not the only lever without human labels. Augmentation quality, initialization, proxy-positive discovery, negative sampling and temperature can change what is learned. Unsupervised mining is possible using proxy pairs; it is not ruled out by the absence of manually labeled queries. For one positive per example in a $B=128$ view pool, there are $127$ negatives and a nominal $\log_2(128)=7$-bit certificate ceiling. The earlier DPR pool has $2B=256$ passages and $255$ negatives, a different nominal ceiling of eight bits; its hard-negative construction is not the simple IID-marginal setting. A uniformly distributed corpus identity among $8{,}841{,}823$ passages has entropy $\log_2(8{,}841{,}823)\approx23.075912$ bits, numerically $16.075912$ above seven. Uniform identity entropy is not the information required for useful retrieval, where several passages may be relevant and queries are not uniform identifiers. The difference does not prove that an affordable batch is incapable of the task.

Activation cost versus a queue

For a particular uncheckpointed half-precision implementation, the cited activation accounting per layer is $s h(34+5as/h)$ bytes, for token length $s$, hidden width $h$ and head count $a$. With twelve layers, define $A(s)=12s h(34+5as/h)\,\mathrm B$. This is implementation-specific accounting, not a universal physical-memory requirement; activation checkpointing, fused attention and replay change it. For $h=768$ and $a=12$:

Retaining $65{,}536$ passage activation graphs by this accounting uses $3{,}401{,}614{,}098{,}432\,\mathrm B=3.401614098432\,\mathrm{TB}$. A width-$768$ fp32 key is $768\times4=3{,}072\,\mathrm B$; the same number of keys is $201{,}326{,}592\,\mathrm B=201.326592\,\mathrm{MB}=192\,\mathrm{MiB}$. The activation/key component ratio is exactly $16{,}896$. This does not compare total training memory: query and current-key activations, gradients, parameters, optimizer state and score matrices remain. Trade-off: Queued keys carry no retained encoder graph, so gradients do not flow through the historical key computations. MoCo also uses momentum-encoder keys rather than recomputing every negative with the current query encoder. This changes the gradient and temporal objective relative to a fully differentiable fresh large batch. Easy negatives can individually receive small softmax weights, but many such terms can have substantial aggregate mass. Their parameter contributions also depend on Jacobians and direction. Stopping their gradients is an algorithmic choice justified by measured training behavior, not a proof that the discarded gradient is negligible.

MoCo timescales: a drift heuristic, not a stability theorem

MoCo enqueues new key vectors and evicts older ones in FIFO order. For momentum $0\le m\lt 1$, the key encoder follows the query encoder by an exponential moving average:

With $B$ new keys per step, queue turnover spans roughly $K/B$ steps, with an indexing-dependent one-batch difference in the oldest age. The EMA weight window $1/(1-m)$ is a useful heuristic. Its steady-state mean weight age is $m/(1-m)$ steps; for $0\lt m\lt 1$, its exponential-decay timescale is $-1/\ln m$. A common design comparison is:

Why monitor drift: Queue contents should remain comparable with current queries. Actual embedding drift depends on the training trajectory, optimizer and normalization, not solely on the EMA weight distribution. The timescale comparison is neither necessary nor sufficient for convergence. With constant encoder parameters, even $m=0$ and a long queue have no representation staleness. Conversely, a numerically long momentum window does not guarantee compatible embeddings during a rapidly changing trajectory. Reported failure: He et al. report failure to converge at $m=0$ in their experiment. That observation supports testing momentum and drift in comparable training conditions; it does not establish a universal impossibility theorem for zero momentum.

Temperature and a bounded-cosine loss floor

For unit-norm vectors with cosine logits divided by $\tau\gt 0$, a positive-minus-negative cosine margin is at most two. With $K$ negatives, the per-example loss therefore obeys:

At $K=65{,}536$, this optimistic floor is approximately $9.090468$ nats for $\tau=1$, and $2.558976\times10^{-8}$ nats for $\tau=0.07$. Simultaneous extreme margins may not be attainable across an entire dataset. For this floor to be no greater than $\varepsilon=0.01$, the exact temperature condition is:

The expression $2/\ln(100K)\approx0.127424855$ replaces $e^{0.01}-1$ with $0.01$ and is a slightly stricter small-loss approximation for the floor. Neither threshold guarantees successful optimization or good retrieval; the floor is only an optimistic necessary feasibility check for reaching that loss. The bounded-margin argument does not apply unchanged to unnormalized dot products. Trying $\tau=0.01$ for sharp distillation is a source heuristic, not the only valid use of that temperature. Failure without this distinction: Loss can reflect the score scale and geometry rather than retrieval quality. Very small temperature multiplies score derivatives by $1/\tau$, can emphasize noisy or false negatives, and needs numerical and validation checks; a lower training loss alone is not evidence of better ranking.

Three negative-pool configurations

Use $P=110\times10^6$, $d=h=768$, twelve layers and twelve attention heads. For a matched unsupervised comparison, both augmented views have $s=128$ tokens. The earlier supervised example instead has $32$-token questions and $128$-token passages; mixing those lengths changes activation and encoder-work comparisons. All memory figures below are scoped activation or vector payloads, not all-in requirements. Configuration 1: ordinary in-batch contrast. With $B=128$ paired views and one positive per anchor, each anchor has $127$ negatives and the nominal certificate ceiling is seven bits, under the earlier sampling assumptions. For the matched two-$128$-token-view setting, retained activations are $2B A(128)=13{,}287{,}555{,}072\,\mathrm B=13.287555072\,\mathrm{GB}$. A different one-passage asymmetric setting gives $B[A(32)+A(128)]=8.0216064\,\mathrm{GB}$. Actual DPR with $B$ questions and $2B$ passages instead gives $B A(32)+2B A(128)=14.665383936\,\mathrm{GB}$ under this same accounting. None alone explains or reproduces the reported eight-V100 deployment; device precision, memory units, optimizer state, partitioning and implementation matter. Configuration 2: fresh large in-batch contrast. At $B_L=65{,}536$, there are $65{,}535$ negatives per anchor and the nominal ceiling is $\log_2(65{,}536)=16$ bits. The batch also processes $512$ times as many anchors as the $B=128$ baseline. Passage-side activation retention is $3.401614098432\,\mathrm{TB}$, requiring at least $\lceil3{,}401.614098432/32\rceil=107$ devices of decimal $32\,\mathrm{GB}$ if that component alone were partitioned perfectly. Retaining both $128$-token views gives $6.803228196864\,\mathrm{TB}$, or at least $213$ such devices before other tensors. These are accounting examples, not unavoidable device counts when checkpointing or gradient replay is available. Encoder work rises $512$ times per step only relative to the same fully trainable two-view baseline; score-matrix work has a different scaling. Configuration 3: MoCo queue. At $K=65{,}536$ queued negatives and $B=128$ current anchors, there are $65{,}537$ logits per anchor, not $65{,}536$. The nominal count term is $\log_2(65{,}537)\approx16.000022$ bits. Queued-vector payload is $201.326592\,\mathrm{MB}$, but stale correlated keys do not automatically satisfy the IID population-bound assumptions. For $m=0.999$, queue turnover spans $512$ steps against the heuristic $1{,}000$-step EMA window, a ratio $1.953125$. The EMA mean weight age is $999$ steps and its decay timescale approximately $999.499917$ steps. These are timescale descriptions, not a guarantee that every key remains compatible. Match the two $128$-token branches before pricing them. The trainable query branch has proxy $6PBs=1.081344\times10^{13}$ FLOPs and the forward-only key branch $2PBs=3.60448\times10^{12}$ FLOPs. Their total is $8PBs=1.441792\times10^{13}$ FLOPs; keys are $25\%$ of that encoder-work proxy. The factor $8/6=4/3$ compares MoCo with the query branch alone, not ordinary fully differentiable contrastive training. At the same $B,s$, two trainable branches have proxy $12PBs=2.162688\times10^{13}$ FLOPs, or $1.5$ times MoCo's encoder work per processed anchor. A fully trainable $B_L=65{,}536$ two-view step has proxy $12PB_Ls=1.107296256\times10^{16}$ FLOPs, which is $768$ times a $B=128$ MoCo step while processing $512$ times as many anchors. Neither ratio establishes cost per learned bit, total training speedup or equivalent gradients. Score matrices, queue operations, replay, communication and optimizer work are outside this encoder proxy. The previous “$384$ times cheaper per bit” comparison mixed baselines and is not justified.

Queue size, memory and measured staleness

The source reports $K=65{,}536$, $B=256$ and $m=0.999$. The turnover span is $256$ steps, with heuristic window/span ratio $3.90625$. At $m=0$, that comparison becomes $256$ steps versus one, but it is not a mathematical violation of a necessary convergence condition. Storing all $8{,}841{,}823$ width-$768$ fp32 keys takes $27{,}162{,}080{,}256\,\mathrm B=27.162080256\,\mathrm{GB}$ and turns over in approximately $69{,}076.742188$ steps at $B=128$. If the chosen policy enforces $K/B\le1/(1-m)$, it asks for $m\ge1-B/K\approx0.999985523347$; rounding up to $0.99999$ satisfies that heuristic, not a universal requirement. A high-momentum encoder can lag adaptation, but it is not necessarily an average of an untrained network: initialization and actual parameter drift matter. Under that same policy, $B=128,m=0.999$ gives the nominal bound $K\le128{,}000$. It is not a hard maximum queue size. Raising momentum to $0.9999$ makes the weight window ten thousand steps and the policy allowance $1.28\times10^6$ keys. The tenfold window change does not by itself prove tenfold embedding drift or acceptable retrieval. Track key/query compatibility, optimization and judged quality.

Practical and experimental boundaries

Evaluate independent crops, inverse-cloze pairs and natural title/body or question/body pairs on the target corpus. ICT removes one sentence occurrence and pairs it with the remainder; words or phrases may recur in that remainder, so zero token overlap is not guaranteed. Cropping likewise does not guarantee a BM25-quality floor. When labels exist, unsupervised pre-training is a candidate initializer. The source reports gains from fine-tuning Contriever rather than raw BERT on MS MARCO and BEIR transfer, and unsupervised Contriever beating BM25 on $\mathop{\mathrm{Recall}}\nolimits @100$ on eleven of fifteen BEIR datasets. Preserve that model/metric scope; the source also reports BM25 as stronger on average for precision-oriented ranking in that comparison. Metric limit: Candidate recall is important before reranking, but assess final ranking, answer correctness and important cohorts too. High $\mathop{\mathrm{Recall}}\nolimits @100$ alone is not equivalent to good top-of-list precision or a final-system guarantee. Deployment may discard the momentum key encoder and build a compatible index with the selected serving encoder. A bi-encoder has the same architectural serving pattern as a supervised bi-encoder, not automatically the same measured latency, vectors or retrieval quality. Version which encoder, pooling, normalization and index are used. GradCache reduces encoder activation retention by caching representations and representation-level gradients, then replaying encoder microbatches. It does not make total memory constant in batch size: cached vectors and gradients scale as $O(Bd)$, and the contrastive score computation may also require a large matrix unless chunked. Exactness relative to the chosen large-batch computation requires correct gradient accumulation and consistent stochastic replay, including dropout RNG and any stateful or batch-dependent layers. Replay adds encoder forward work, roughly an extra forward pass in the basic scheme; it does not erase large-batch arithmetic or make a stopped-gradient queue equivalent to that batch.

20.5 Cross-lingual and low-resource retrieval

Why lexical retrieval returns zero

For ordinary lexical BM25 without expansion or extra scoring features, an empty analyzed term overlap makes the query-document score sum zero. If every candidate has empty overlap, ranking is determined by a tie policy rather than semantic evidence. An Arabic query and English passages can have this failure, but mixed-script names, codes and analyzer behavior can also create overlap; zero is not inevitable for every language pair. Failure of parameter tuning alone: Changing length normalization or term-frequency saturation cannot supply a missing analyzed term. Translation, multilingual expansion, learned lexical representations or a separate dense path change the available matches and therefore the premise. Keep a lexical route for native-language exact matches and language-independent identifiers where it helps. Do not assume that every cross-language query lacks overlap or that a Latin-script match is necessarily the right answer. Preserve source-language text, analyzer settings and access constraints.

Language centroid model

The following is a toy unit-norm embedding decomposition, not a claim that a real multilingual encoder has a separable language subspace. For text $t$ in language $l$:

Assume unit language directions $\mu_l$, unit semantic residuals $r(t)$ orthogonal to every language direction, a common amplitude $\alpha$, and $\mathbb E[r(t)\mid l]=0$. Then $\mathbb E[e(t)\mid l]=\alpha\mu_l$, not $\mu_l$. The language component has norm $\alpha$ and squared norm share $\alpha^2$. For query language $a$ and document language $b$:

For a cross-language positive, let centroid-direction cosine $c=\mu_a^{\mathsf T}\mu_b\in[-1,1]$, semantic score $s^+$, and a same-query-language distractor have semantic score $s^-$. If $\Delta s=s^+-s^-\gt 0$, the positive outranks that distractor exactly when:

Equivalently:

For $s^+=0.9$, $s^-=0.1$ and $c=0.5$, the exact squared-amplitude threshold is $0.8/1.3\approx0.615384615$, so $\alpha\lt \sqrt{0.8/1.3}\approx0.784464541$. At $\alpha=0.9$, these specified examples score $0.81(0.5)+0.19(0.9)=0.576$ for the cross-language answer and $0.81+0.19(0.1)=0.829$ for the same-language distractor. They are the given scores, not global best/worst bounds. For $c=0$, the amplitude threshold is exactly $2/3\approx0.666666667$, not an exact cutoff of $0.67$. Model-specific failure: At $\alpha=0.9,c=0.5$, even a cross-language residual score of one gives at most $0.81(0.5)+0.19=0.595$, while a same-language residual score of minus one gives at least $0.81-0.19=0.620$. Thus no residual semantic gap can overcome language in this particular toy case. More generally $\Delta s\le2$; this is not a universal diagnosis of multilingual encoders or proof that centering fixes them.

Mean-centering and sampling fixes

Under the assumptions, the true language mean is $\bar e_l=\alpha\mu_l$. Its norm identifies $\alpha$ and, when means are nonzero, their cosine identifies $c$. A thousand documents per language is a source sampling heuristic, not a precision guarantee; topic mixtures, domain imbalance, sample error and drift can confound the estimates. If $\alpha=0$, means vanish and their cosine is undefined even though there is no language component. Subtract the actual mean $\bar e_l$, not $\alpha$ times that mean again. Centered vectors are $\sqrt{1-\alpha^2}\,r(t)$, whose dot product is $(1-\alpha^2)s$. With the common nonzero residual scale assumed here, renormalizing centered vectors makes cosine exactly $s$. Real encoders can have unequal amplitudes, residual norms and topic-dependent means; subtraction can hurt useful signal or produce near-zero vectors and requires held-out evaluation. The source reports improved cross-lingual sentence retrieval after centroid subtraction on mBERT and describes LAReQA as testing cross-language alignment beyond translation-pair ranking. Language-controlled negatives can remove one shortcut in the toy model. Draw them from the positive passage's language, which need not be the query's language. Then the centroid term is the same for positive and negative. An eighty-percent-English negative pool can offer a language shortcut when the positives are minority-language and language matches reveal the label; the corpus proportion alone does not prove this for every query-positive pair. Controlled sampling does not guarantee that all remaining cues are semantic or that false negatives disappear.

Capacity and tokenization

The source reports XLM-R training on $2.5\,\mathrm{TB}$ across one hundred languages and describes capacity/resource competition as the curse of multilinguality. Its outcome depends on parameter budget, data allocation, vocabulary, languages and tasks. An eight-language model does not mathematically have to outperform a hundred-language model on those eight at equal parameter count; compare them with matched training resources and target-language evaluation. Fertility is tokenizer tokens per word. Under constant mean fertilities, a $512$-token window corresponds to $512/1.3\approx393.846154$ English words or $512/3\approx170.666667$ Swahili words. For equal word volume and negligible boundary losses, the token/chunk-volume ratio is $3/1.3\approx2.307692$. Swahili context per window is $1.3/3\approx43.333333\%$ of English context in this model. Do not divide rounded word counts and call the result exact. Real chunk counts depend on length distributions, overlap, truncation and document boundaries. Use meaningful boundaries when possible, then enforce the model's actual tokenizer limit and preserve source metadata.

Translation placement and latency

Assume twenty million documents across eight equally sized language partitions, a nominal $25$-token query, and a $300\,\mathrm{ms}$ request-$p_{95}$ target. Translation is priced at an assumed $\$1.00$ per million output tokens with a nominal autoregressive decode rate of $20\,\mathrm{ms/token}$. For the worked prices, stipulate $250$ output tokens per translated document and $25$ output tokens per query translation; source input lengths alone do not imply those output lengths. These are scenario assumptions, not measured latency distributions or a verified vendor quote. Configuration 1: Translate the corpus to English offline. If all twenty million documents incur a $250$-output-token conversion, the volume is $5\times10^9$ output tokens and the one-time charge is $\$5{,}000$. Uniform five-percent monthly churn gives one million conversions, $250\times10^6$ output tokens and $\$250$. If one of eight equal partitions is already English and identity translation is skipped, instead translate $17.5$ million documents: $4.375\times10^9$ output tokens costing $\$4{,}375$, with $\$218.75$ monthly under uniform churn. Translation expansion, retries and input-side billing can change the invoice. A non-English query producing $25$ English output tokens has nominal decode component $25(20\,\mathrm{ms})=500\,\mathrm{ms}$ and output-token charge $\$0.000025$. Adding a stipulated $10\,\mathrm{ms}$ search stage gives a nominal $510\,\mathrm{ms}$ path, excluding translation prefill, time to first token, networking and queues. An English query need not take this translation path. This deterministic sum is not the request $p_{95}$. Configuration 2: Keep eight native indexes, nominally $2.5\times10^6$ one-vector documents each, and translate the query to the seven other languages in parallel. Chunking and language imbalance can change those counts. For seven fully concurrent translations with the same stipulated output length/rate, nominal decode critical path is their maximum, $500\,\mathrm{ms}$ in this deterministic example, not seven times that. Output-token cost is seven times the single-target cost, $\$0.000175$ per query. If eight searches and fusion together add a stipulated $15\,\mathrm{ms}$, the nominal path is $515\,\mathrm{ms}$; real scheduling and resource contention need measurement. Tail probabilities do not inherit an individual service's percentile. If seven translation times were independent and identically distributed, all seven finish by their individual $p_{95}$ with probability $0.95^7\approx0.698337$, not $0.95$. For their maximum to meet a ninety-fifth-percentile target, the corresponding individual quantile would be $0.95^{1/7}\approx0.992699$. Dependence and heterogeneous paths change that calculation. Do not add individual component percentiles to claim an aggregate request percentile. Configuration 3: Use one multilingual bi-encoder and one index. For one multilingual query encoder, the forward proxy is $2(110\times10^6)(25)=5.5\times10^9$ FLOPs. At a stipulated effective $50\times10^{12}\,\mathrm{FLOP\,s^{-1}}$, division gives $0.11\,\mathrm{ms}$. Assign a nominal $10\,\mathrm{ms}$ encoding-service time inclusive of that arithmetic, plus $10\,\mathrm{ms}$ search, to obtain a nominal $20\,\mathrm{ms}$ path. Calling the first ten milliseconds additional overhead and then ignoring the arithmetic would be inconsistent. The nominal ratios are $510/20=25.5$ and $515/20=25.75$, not matched measured speedups or a demonstrated $p_{95}$ guarantee. This design has no online translation fee, not zero serving cost. With exactly one width-$768$ fp32 vector per document, payload is $(20\times10^6)(768)(4)=61.44\times10^9\,\mathrm B=61.44\,\mathrm{GB}$ before graph links, identifiers, text, replicas and runtime buffers. Claim limit: Tokenization, fertility, translation expansion and chunk boundaries can change vector counts. Corpus payload and a nominal encoding/search sum neither establish all-in memory fit nor prove either the original $300\,\mathrm{ms}$ or tightened $150\,\mathrm{ms}$ request-$p_{95}$ target.