Chapter 22: Reranking

00-START-HERE · 01-READING-ROUTES · 21_learned_sparse_and_multi_vector_retrieval · 23_generative_retrieval This chapter explains why Retrieval-Augmented Generation (RAG) needs a second ranking stage, how to price and size it, and how cross-encoders, large language model (LLM) ranking modes, ranking losses, sliding-window prompting, instruction-aware scoring, and document-bound versus model-bound diagnosis support that choice.

TL;DR

• A fixed-candidate reranker cannot retrieve omitted evidence. The exact recall product needs a second-stage probability conditioned on first-stage inclusion.
• Compute, serial dependencies and hardware capacity are different budgets. The fuller BERT proxy is $0.195346\,\mathrm{ms}$ per pair at the assumed rate, not a measured response time.
• Choose depth from measured first-stage coverage, conditional reranking quality and cost. A fixed IID distractor model illustrates a possible trade-off, not inevitable collapse.
• Pointwise, pairwise and listwise LLM modes trade tokens, comparisons, dependencies, calibration and order sensitivity. Unlimited parallelism cannot be assumed for free.
• Listwise loss couples same-query candidates; pointwise classification is not inherently incapable of ranking. Negative sampling and explicit probability/logit conventions matter.
• Tail-to-head overlapping windows give deep candidates a promotion path in this one-pass schedule. Call counts are integers, and rescue probabilities require conditional hand-off assumptions.
• Instruction-aware scoring needs informative supervision and explicit intent access, not necessarily one particular architecture. Symmetry-based blind-model floors require a fixed balanced candidate scenario.
• Compare one more document with a specific stronger ranker at the current context size, including costs. The toy crossover near five is not a universal pipeline boundary.

The story

Think of retrieval as auditions for a small theater. The casting assistant invites candidates; the director can only choose among those invited. Keeping fewer, better-ordered passages can save generator work, but the saving and quality gain must be measured. Pointwise auditions score one actor, pairwise auditions compare two, and listwise auditions order a group. Some decisions run independently, yet a single theater has finite staff: many independent auditions are not one audition's wall-clock time at no extra cost. A tail-to-head relay lets a deep candidate move through overlapping groups; a disconnected single-pass schedule cannot move it across block boundaries. Instructions are role notes. On-topic candidates that differ in whether they satisfy those notes help reveal whether the director actually uses them. Finally, compare the next extra context passage with a named better director at today's cast size and budget. The useful next investment can change with the operating point; neither a toy threshold nor one ablation determines it forever.

Decoder table

Core mechanics

Mathematical conventions

Use decimal storage units: $1\,\mathrm{GB}=10^9\,\mathrm B$ and $1\,\mathrm{TB}=10^{12}\,\mathrm B$; fp32 stores $4\,\mathrm B$ per coordinate. One multiply-accumulate counts as two FLOPs. Keep full precision until reporting, and use $\approx$ for rounded results or a model approximation. A percentage-point change is absolute; relative change divides by a stated baseline. For the worked compute scenarios, $P=110\times10^6$ encoder parameters, $P_g=7\times10^9$ generator/LLM parameters, $F=3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$ sustained throughput per assumed accelerator and $B_{\mathrm{mem}}=2.0\times10^{12}\,\mathrm{B\,s^{-1}}$ memory bandwidth. $2PL$ is a linear forward-work proxy, not a measured latency. The fuller encoder proxy adds $4\ell L^2h$ for attention matrix products. Numerical pair lengths $288$ and $320$ omit extra special/formatting positions unless already counted. Biases, normalization, activation functions, output selection, transfers, queues and many kernel costs remain outside these proxies. Work divided by throughput is an idealized resource-time estimate, not a latency or p99 guarantee. Independent calls do not all finish in one call-time on one fixed-throughput device. An unlimited-resource critical path and total work are separate quantities; batching does not create extra aggregate throughput. The flat-read estimate assumes the stated bytes actually stream through the bandwidth bottleneck and is not an upper bound on first-stage latency. Generator prefill examples count only document tokens and omit attention-length effects, fixed prompt work and decode. The recall product uses nested events for one specified gold evidence item, on the same query population. $R_2$ is conditional on that item being retrieved, not an independently averaged recall. Multi-document evidence, multiple golds and macro averaging need their own denominators. The distractor model assumes a single gold, no score ties and IID outranking indicators with fixed error probability; real score dependence and query heterogeneity can violate it. For losses, $z_i$ is an unbounded relevance logit, $r_i=\sigma(z_i)$ a binary probability, and $\pi_i=e^{z_i}/\sum_j e^{z_j}$ a within-list softmax probability. Do not feed binary probabilities into a logit formula without defining that change. All cited empirical scores and configurations remain reported source inputs. The geometric, window-survival and instruction-symmetry examples are conditional models, not calibrated production laws.

22.1 Why a second stage exists at all

What

Let $G$ denote inclusion of one specified gold item in the candidate set and $H$ its inclusion in final context. Because fixed-set reranking selects only from those candidates, $H\subseteq G$:

This is a set-membership ceiling for a fixed candidate pool, with $1\le m\le k\le N$. If $R_1(k)=0$, final gold inclusion is zero and the conditional factor need not be defined. For a micro-average over gold items, sample/weight the same evidence population in both factors. Multiplying separately macro-averaged per-query recalls generally does not give macro end-to-end recall. Candidate recall also does not determine answer quality or context-order effects.

Why

A dense first-stage dot product costs

Use the assumed $P=110\times10^6$ encoder parameters, $32$ query positions and $256$ document positions. The simple $2PL$ proxy, before the attention correction, is:

One cross-encoder pair is

times the dot-product work in this proxy, not the measured latency. Exhaustive cross-encoding over $N=10^7$ passages would use:

The staged system with $k = 100$ costs

The work-proxy reduction is $6.336\times10^{17}/(6.35136\times10^{12})\approx99{,}758.162031$. Stage two scores $100/10^7=10^{-5}$ of the corpus and uses approximately $99.758162\%$ of this retrieval arithmetic. The source reports that moving answer-bearing text within context can change generation quality. A set-recall number cannot measure that positional effect, and a relevance sort is not automatically the generator-optimal ordering.

Failure without it

If the specified evidence is missing from the candidate set, a fixed-set reranker cannot recover it. If it was retrieved but cut before the final $m$, inspect ranking/truncation, though first-stage candidate composition can still affect difficulty. End-to-end accuracy alone does not separate those failures, and a generator can sometimes answer without that particular gold chunk. Passing every candidate avoids this truncation but increases context work and can change generation quality; it is not guaranteed to bury or harm the answer on every query.

Cost, complexity, and worked example

For this resource scenario use $N=10^7$, document length $256$, embedding width $d=768$ in fp32, the stated BERT and generator proxies, and the common throughput/bandwidth assumptions. Flat vector payload is:

Dividing by $2.0\,\mathrm{TB/s}$ gives $15.36\,\mathrm{ms}$ of idealized streaming time. Dot work alone gives approximately $0.045176\,\mathrm{ms}$ at the stated FLOP rate, so the larger roofline component is memory here. This is not a first-stage upper bound: query encoding, selection, nonideal bandwidth, index traversal and queues can add cost. ANN may read fewer vectors but can also have irregular accesses and other overhead. The following partial-time scenarios assume sequential stages and the idealized flat-read component; they are not measured request latencies.

The $100$-candidate reranker costs

Reducing the document prefill from one hundred passages to five removes approximately $1{,}001.411765\,\mathrm{ms}$ under the linear generator proxy, or $53.737374$ units of resource time per unit of the $18.635294\,\mathrm{ms}$ linear rerank estimate. Adding reranking to raw top-five increases this partial-time baseline by $27.378319\%$. Two hundred passages contain $51{,}200$ document tokens, with proxy $7.168\times10^{14}$ FLOPs and $2{,}108.235294\,\mathrm{ms}$; linear reranking of those candidates is $37.270588\,\mathrm{ms}$. Actual long-context attention and serving costs can differ. The source reports DPR hit rates $78.4\%$ at depth twenty and $85.4\%$ at one hundred: the $7.0$ percentage-point ceiling increase is $32.407407\%$ of the former $21.6$-point retrieval miss rate, not a guaranteed answer-quality gain.

Decisions and limits

• Measure candidate coverage and final context inclusion on the same held-out population, with explicit gold and averaging rules.
• Compare reranking cost with generator work saved and the final answer-quality change, not only one latency component.
• Use both set and rank metrics, plus generation tests where context order matters.
• Treat depth one hundred as a starting heuristic; benchmark actual tails and capacity.
• Recompute dollar savings under actual token/call billing, caching and fixed serving costs.
• Cached prefill or position-insensitive generation can reduce some benefits, but neither proves that all ranking, diversity or noise-filtering value disappears.

22.2 Cross-encoders and choosing rerank depth

What

A cross-encoder forms a joint sequence such as [CLS] query [SEP] document [SEP]. Its final document token states depend on the current query. Those joint states cannot be reused as a query-independent document embedding in this baseline, although text, tokenization and other preprocessing can be cached. Joint interaction can improve ranking but does not mathematically guarantee a quality win over every bi-encoder.

Why

Let $L=L_q+L_d+L_{\mathrm{special}}$. The worked numbers set additional special positions to zero; an actual tokenizer may add them. With layer count $\ell$ and width $h$, define the fuller proxy:

For BERT-base with $P = 1.10 \times 10^{8}$, $\ell = 12$, $h = 768$, and $L = 288$, the linear term is

The attention term is

The total proxy is $66{,}417{,}647{,}616$ FLOPs per pair; attention matrix products are $4.825833\%$ of the linear term. At $L=1{,}024$, linear and quadratic terms are $225{,}280{,}000{,}000$ and $38{,}654{,}705{,}664$ FLOPs. The total increases by $3.973864$ for a length factor $3.555556$. A two-point power-law interpolation gives exponent approximately $1.087684$; this is a fitted description over those lengths, not a universal transformer exponent. Actual pair shape and implementation matter.

At fixed lengths, total resource work scales with depth, giving $0.195346k\,\mathrm{ms}$ at this rate. Fit the reported two endpoint hit rates exactly: $b_D=(0.854-0.784)/\log_2(100/20)\approx0.030147359$ and $b_B=(0.737-0.591)/\log_2(100/20)\approx0.062878777$ recall fraction per doubling. The dense fit gains $3.014736$ points from depths twenty to forty for $3.906920\,\mathrm{ms}$ additional work, or $0.771640$ points per resource millisecond; from one hundred to two hundred the corresponding ratio is $0.154328$. The BM25 fit reaches $78.4\%$ around continuous depth $167.884612$, so integer depth $168$ is needed under that interpolation, about $8.4$ times dense depth twenty. This is an extrapolation beyond the fitted BM25 endpoint, not a measured guarantee.

Failure without it

To illustrate distractor accumulation, condition on one specified gold being in the pool. Assume each of the other $k-1$ candidates is irrelevant and independently outranks it with the same fixed probability $\epsilon$, with no ties. Then the number above gold is:

with approximation

In this toy, $R_2(m\mid k)=\Pr(X_k\le m-1)$. A shared random gold score can correlate outranking events, and hard negatives need not have a common error probability. Multiple relevant passages, listwise context effects and query-dependent difficulty change the model. With fixed IID error, increasing distractors lowers this conditional survival while a fitted first-stage curve rises; the product can have an interior maximum. Neither monotonic conditional degradation nor collapse is a universal empirical law. The population conditioned on retrieval can change with depth, as can scores and error distributions. Redeem a stronger first stage as lower depth only after validating the resulting quality and costs.

Cost, complexity, and worked example

Use $m=5$, $\epsilon=0.01$ and dense hit-rate fit $\widehat R_1(k)=0.784+b_D\log_2(k/20)$ over the displayed range. Retain the unrounded endpoint slope. The table uses the exact Binomial conditional CDF, not rounded Poisson values. A $20\,\mathrm{ms}$ proxy-work budget permits at most $\lfloor20/0.1953460224\rfloor=102$ candidates. At depth one hundred the reported BM25 and dense candidate ceilings are $73.7\%$ and $85.4\%$; dense depth twenty has a $78.4\%$ ceiling. Candidate ceilings are not final answer accuracies.

The exact displayed model has an unconstrained integer maximum at $k=137$, inclusion probability approximately $85.706257\%$, at $26.762405\,\mathrm{ms}$ of proxy work. That is outside the stated budget. Within $k\le102$, it selects $k=102$, approximately $85.179939\%$. The former “near one hundred fifty” observation was only approximate and did not enforce the budget. Values above depth one hundred extrapolate the first-stage fit; the model is not a production forecast. The source reports BERT-large reranking at depth one thousand with MS MARCO $\mathop{\mathrm{MRR}}\nolimits @10=0.365$, versus $0.187$ for BM25. This does not identify a distractor rate. At $\epsilon=0.005$, the exact toy has top-ten inclusion $0.968714724$ but truncated expected reciprocal rank only $0.195975404$, even with first-stage recall one. Its untruncated expected reciprocal rank is bounded by $[1-(1-\epsilon)^k]/(k\epsilon)\approx0.198669206$, so it cannot simultaneously explain MRR $0.365$. A first-stage miss factor only lowers that number. Matching $0.365$ under this particular toy with recall one would need approximately $\epsilon=0.002519761$, not an empirically established rate. In general, $\mathop{\mathrm{MRR}}\nolimits @10=P(\mathrm{rank}\le10)\,\mathbb E[1/\mathrm{rank}\mid\mathrm{rank}\le10]$. The harmonic-mean rank among successes is that success probability divided by MRR, not simply one divided by MRR unless every query succeeds. The source's Natural Questions and MS MARCO figures are not one matched calibration experiment.

Decisions and limits

• Start with a tractable depth such as one hundred, then measure candidate coverage, conditional rank metrics and actual latency tails after each first-stage change.
• Evaluate several depths on the same held-out queries; attach uncertainty and important cohorts.
• Measure full gold-rank/error distributions. A single mean outranking rate is not enough to validate IID Binomial tails.
• Compare a better first stage with recurring rerank work under a common quality and cost target.
• Recompute pair lengths when truncating. A shorter selected window can save work but can also remove needed evidence, and finding the best window is not free.
• Validate early exits, token policies and long-evidence cases; neither an empirical layer choice nor the two-point length exponent is a universal guarantee.

22.3 LLM rerankers: pointwise, pairwise, listwise

What

Pointwise prompts return a candidate score, pairwise prompts compare two candidates, and listwise prompts order a group. Classical learning-to-rank can put coupling in the training loss without placing multiple documents in one forward pass. RankNet is one such pattern; the source also discusses McRank and SetRank. Without order-dependent features, self-attention produces permutation-equivariant per-document scores, not an invariant score vector: permuting inputs permutes outputs. A resulting document order can be invariant after mapping scores back, subject to ties and implementation. A frozen LLM can sometimes supply useful pointwise rankings; joint comparison is an option to test, not a logical necessity or a guarantee of calibration.

Why

For a $7$-billion-parameter reranker under the $2P$ rule and the stated throughput, one input token costs

With query/instruction preamble $32+50=82$ tokens and $256$ document tokens, pointwise input is $338$ tokens and pairwise input $594$, with work proxies $13.917647$ and $24.458824\,\mathrm{ms}$ per call. Their verdict output tokens are omitted. For $w=20$, listwise prefill is $5{,}202$ tokens and the assumed identifier output is $3w=60$, totaling $5{,}262$ and $216.670588\,\mathrm{ms}$. This deliberately applies one FLOP rate to prefill and serial decode; actual decode throughput, identifier lengths and formatting differ. Call counts are:

and, at stride $\frac{w}{2}$,

For $k=100,w=20$, that aligned schedule has nine dependent calls. Pairwise top-$m_h$ extraction need not compare all pairs. For a deterministic transitive comparator, Floyd heap construction plus repeated extraction has a conservative comparison bound:

At $k=100,m_h=10$, the bound is $340$ comparisons versus $4{,}950$ all-pairs comparisons. It is not a fixed observed count. Extraction and heap construction both have dependencies; the extraction term $140$ alone is not the full serial depth. A balanced tournament uses $k-1=99$ comparisons for a winner and at most $\lceil\log_2k\rceil=7$ rounds, with adequate parallel workers and explicit byes. A cyclic, stochastic LLM comparator need not define a total order, so different schedules can yield different results.

Failure without it

Pointwise scores need validation for ranking and cross-query confidence; a prompted scalar is not automatically calibrated. Pairwise cycles or inconsistent repeated judgments can make a comparator-based algorithm return an unstable order or fail its assumed ordering contract. Listwise outputs can depend on input permutation, context length and formatting. Declaring candidates together does not remove positional bias. The source treats the origin of zero-shot comparative ability as a hypothesis, not a proved result.

Cost, complexity, and worked example

For these specific input/output counting rules and windows, pointwise uses the least modeled LLM work, not universally the cheapest LLM ranking design. Its work is approximately $71.246125$ times the fuller BERT proxy. The parameter-and-input-length-only ratio is:

Dividing that ratio by the full attention correction $1+3{,}057{,}647{,}616/(6.336\times10^{10})$ gives approximately $71.246125$. Do not explain a difference between rounded inputs as measured overhead. These resource-time totals are neither invoices nor measured response times. Independent calls permit concurrency only when hardware and service limits support it. For total work $W$ on $G$ identical resources, $W/(GF)$ is a work-capacity constraint under the assumed rate; the model-call dependency path is another constraint. Giving a hundred pointwise calls one-call wall time implicitly supplies a hundred full-rate workers in this idealization. Batching can improve utilization but cannot increase fixed aggregate capacity without bound. A cascade reranks one hundred candidates to twenty with the fuller cross-encoder proxy, then uses one listwise call. Its modeled sum is $19.534602+216.670588\approx236.205190\,\mathrm{ms}$ before request overhead, about $8.255684$ times below the full nine-window work sum, with nine times fewer LLM tokens. Add a stipulated $80\,\mathrm{ms}$ round trip for the LLM call and it becomes $316.205190\,\mathrm{ms}$. The cascade may discard candidates the LLM would have rescued. Under the simplified total context budget:

At $w=20$, this simplified prompt/output count uses $5{,}262$ of $8{,}192$ tokens, leaving $35.766602\%$ of capacity. Eleven additional passage slots are $55\%$ of twenty used slots, not fifty-five percent of the context budget. Actual identifiers, separators, instructions and model output reserves can lower capacity. Quality and ordering reliability can bind before nominal fit; raw fit alone does not prove the limiting factor.

Decisions and limits

• Benchmark frozen pointwise, pairwise and listwise prompts against a trained cross-encoder; treat any default as a starting heuristic.
• Count actual tokens, model calls, dependency rounds and available worker capacity. Report both work and request latency.
• A cross-encoder cascade can reduce LLM work but creates a candidate ceiling; evaluate disagreements and gold survival.
• Validate pointwise calibration where needed, pairwise cycles/repeatability and listwise permutation sensitivity and repair rates.
• Larger all-pairs or repeated-permutation designs can be useful offline, but hardware cost remains even when user latency is irrelevant.

22.4 monoT5 to RankT5: from classification to ranking losses

What

monoT5 prompts a query/document pair with Relevant: and uses the output-token logits for true and false. The renormalized probability over those two tokens, rather than their raw vocabulary probability, is:

The source reports monoT5 transfer from MS MARCO training to TREC Robust04, and describes RankT5 using a relevance logit associated with the sentinel extra_id_10. Reported improvements depend on the model variant, loss and training recipe; separable classification is not a theorem that ranking must fail. For the workload arithmetic below, separately assume $Q_{\mathrm{train}}=500{,}000$ query groups. That is not interchangeable with five hundred thousand individual query-document pairs.

Why

The monoT5 pointwise objective is

Its per-logit derivative is $r_i-y_i$, while shared parameters can still couple the learned representations. Compare two-candidate lists: a negative binary probability changes $0.02\to0.10$, while the positive is either $0.90$ or $0.05$. BCE on the negative has the same increase in both cases:

For the one-positive illustrative listwise loss, convert those binary probabilities to logits $z_i=\ln[r_i/(1-r_i)]$, then apply a candidate softmax. RankT5 can learn unbounded logits directly; the conversion here only makes the numerical comparison fully specified:

With exactly one positive and the changing negative, listwise loss increases by $0.010005086$ nats for gold probability $0.90$, and $0.807292526$ nats for gold probability $0.05$, a ratio $80.688215$. The earlier $0.5859$ and $58.8$ figures did not follow from these stated probabilities. Adding other candidates changes the denominator and these values; they are not universal penalties. For a one-hot target:

The positive logit has pull magnitude $1-\pi_t$ and all negative logits together have equal opposing mass. Individual negatives receive weights $\pi_i$. With summed unweighted BCE, the negative-to-positive logit-gradient mass is instead:

At positive probability $0.90$ and all $k-1$ negatives $0.02$, that mass ratio is $3.8$ at $k=20$ and $19.8$ at $k=100$. Class weighting or separately normalized positive/negative terms changes it. Neither gradient balance nor coupling alone proves better held-out ranking.

Failure without it

High classification accuracy can coexist with poor ranking, but calibrated relevance probabilities can also produce useful rankings. A strictly increasing within-query map preserves all score order and ties, so it cannot change MRR, NDCG or MAP if tie handling is unchanged. Converting a sigmoid to its logit has this property away from numerical saturation. Positive-slope Platt scaling does too; isotonic regression is only nondecreasing and can merge distinct scores into ties, allowing metric changes under a tie policy. A negative-slope map can reverse order. Calibration is not a replacement for discriminative ranking supervision. For the same gain difference and ideal-gain normalizer, full-depth DCG/NDCG discount gaps are $1-1/\log_2 3\approx0.369070246$ between ranks one/two and $1/\log_2 100-1/\log_2 101\approx0.000324515$ between ranks ninety-nine/one hundred. Their ratio is about $1{,}137.299329$. Actual NDCG swap changes also multiply the gain difference and divide by IDCG. At cutoff ten, both tail ranks are outside the metric and their swap has zero effect; the quoted finite ratio does not apply there. LambdaRank can weight score-pair updates by the relevant metric change, under its own conventions.

Cost, complexity, and worked example

The $32{,}128$-entry language-model head costs

A full vocabulary projection is approximately $0.094938\%$ of the encoder-only proxy below, so changing only a selected output row offers a small arithmetic saving. It does not follow that the complete encoder-decoder scoring step is that encoder cost alone. For the explicit width-$768$, feed-forward width-$3{,}072$, twelve-layer encoder scenario:

At $L=288$, encoder linear work is $48{,}922{,}361{,}856$ FLOPs and attention matrix products add $3{,}057{,}647{,}616$, giving $W_{\mathrm{enc}}=51{,}980{,}009{,}472$. Embedding lookups, norms, biases and nonlinearities are not counted. A full encoder-decoder variant also needs document-dependent cross-attention keys/values for its single decoder step. They cannot be assumed already cached for a newly scored candidate. With twelve decoder layers of the same width and feed-forward size, their projections alone cost $4(12)(288)(768)^2=8{,}153{,}726{,}976$ FLOPs. Other one-position decoder linear maps cost $2(12)[6(768)^2+2(768)(3{,}072)]=198{,}180{,}864$ FLOPs; self/cross attention dot/value products add $10{,}653{,}696$, and the full vocabulary head adds $49{,}348{,}608$. This explicit decoder total is $8{,}411{,}910{,}144$, approximately $16.182972\%$ of encoder work, not the earlier under-one-percent claim. Use $W_{\mathrm{T5}}=60{,}391{,}919{,}616$ FLOPs, or $0.177623\,\mathrm{ms}$ at the assumed rate, for the following matched full encoder-decoder workload. This is a stated architecture/caching scenario, not an externally verified implementation profile. An encoder-only variant, projected/cached memory or a selected-head-row implementation has another budget. Changing the loss while preserving the scoring architecture need not change per-candidate serving work; it does not imply all model variants have identical cost.

The eightfold training-work ratio comes from equal query-group epochs with two versus sixteen scored candidates and the same assumed $3W_{\mathrm{T5}}$ training work per candidate. It is not an unavoidable eightfold measured training-time penalty or a conclusion from a pair count. List construction, optimizer state, backward work, batching and hardware are additional factors. For a matched sixteen-candidate loss example, let the positive binary probability be $0.90$. Fifteen random negatives with probability $0.001$ give $\pi_t\approx0.998334444$ and list loss $0.001666945$ nats after the defined logit transform. Replace one by a $0.92$ hard negative and retain fourteen at $0.001$: the loss becomes $0.823883686$, about $494.247709$ times larger. Uniform sampling from a large corpus does not by itself establish any of those predicted probabilities. In the separate twenty-document gradient figure, probabilities are $0.90$ for gold, $0.92$ for one distractor and $0.02$ for eighteen tails. Summed BCE allocates $71.875\%$ of negative logit-gradient mass to the top distractor; softmax list loss allocates $96.904557\%$. Neither number is a fraction of encoder-parameter gradient norm. Subtracting the source's rounded BERT embedding count gives $86.2\times10^6$ nonembedding parameters, approximately $1.489785\%$ above the simple T5 encoder-block count. Embeddings are about $21.636364\%$ of the stated BERT total; a percentage markup over a smaller nonembedding denominator is a different quantity. These bookkeeping differences do not isolate a measured architecture-latency cause.

For a one-hot target and the same softmax logits, the added-term gradient equals $\varepsilon_{\mathrm{poly}}\pi_t$ times the cross-entropy gradient. If $\varepsilon_{\mathrm{poly}}\ge0$, the combined gradient is $(1+\varepsilon_{\mathrm{poly}}\pi_t)$ times it. The ratio can be large near high confidence, but both terms vanish as the example becomes perfectly classified. This identity is not a guarantee of improved generalization or a reason to ignore negative quality.

Decisions and limits

• Compare pointwise, pairwise and listwise losses under controlled candidate lists and evaluation, rather than call one universally correct.
• Train on useful competing candidates from refreshed retrieval pools; account for false negatives and label uncertainty.
• Define query-group count, list size, epoch and loss normalization before quoting a training multiplier.
• Keep the scoring architecture fixed before claiming unchanged serving work; include decoder memory projections if applicable.
• Calibrate cross-query abstention separately. A pointwise loss is neither necessary nor sufficient for calibrated confidence, and a ranking loss can be combined with calibration.
• Test any poly-1 coefficient after the base objective and negative sampling are sound.

22.5 Zero-shot listwise prompting and the sliding window

What

A prompted listwise ranker returns a permutation of local passage identifiers; it does not return calibrated relevance scores. For this one-pass progressive schedule, start with the tail window, rank it in place, and shift toward the head by $1\le s\le w$. For an implementation that clamps the last full window to start at rank one:

For $k=100,w=20,s=10$, the count is nine. When $k\le w$, use one window over the available candidates. If $(k-w)/s$ is not an integer, the last shift is shorter under this policy; an implementation using partial windows can have different token counts.

Why

For aligned full windows, the top $w-s$ positions of a ranked window overlap the next one; its bottom $s$ never get revisited in this pass. A candidate initially at rank $g$ first enters a particular window and, if retained in every overlap, can reach the final head window after:

The default counts are nine participating calls from rank one hundred, five from fifty, two from twenty and one from ten. To finish at rank one, the required events are intermediate overlap survival and then winning the final window; merely surviving that final window is not enough. Let $p_t$ be each required event's probability conditional on all earlier successes. The chain rule is:

At an assumed constant conditional value $p_{\mathrm{window}}=0.9$, tail and head examples give $0.9^9\approx0.387420489$ and $0.9$, ratio $2.323057$. The schedule determines the number of required hand-offs, not their probabilities. Different window content, position bias and correlated model errors can change those conditional values. This is one progressive block-ranking pass, not a guarantee of a globally sorted list.

Failure without it

Five disjoint windows of size twenty can promote a rank-one-hundred candidate only to rank eighty-one in that pass. A single head-to-tail overlapping sweep has the same limit for that candidate because it appears only in the last window. These facts do not rule out other algorithms, repeated passes, tournaments or global comparisons. Parse permutation output explicitly: retain valid identifiers in first-appearance order, remove duplicates/out-of-range values and append missing identifiers in their original relative order. On total parse failure, fall back to the input order and log the failure; repair can conceal an ineffective ranker if not monitored.

Cost, complexity, and worked example

The worked budget assumes $82+256w$ input tokens and $3w$ output identifier tokens per full window, ignoring additional input labeling/formatting. It applies the same forward-FLOP rate to both phases. Thus total modeled tokens per call are:

Add a separately stipulated $80\,\mathrm{ms}$ request/queue overhead per call. The windows depend on earlier outputs, so those overheads cannot be hidden by parallelizing windows within the same pass. The compute column remains a work proxy, not measured decode latency.

At stride $\frac{w}{2}$, document slots total

The default uses $180$ document slots, $46{,}080$ document tokens and $1{,}278$ preamble/output tokens, totaling $47{,}358$. Changing to $w=40,s=20$ gives four calls and $41{,}768$ modeled tokens: an $11.803708\%$ token saving. With the stipulated round trips, time-proxy saving is $23.601803\%$. Wider windows alter both content and the number of hand-offs. Under separate constant-conditional-success assumptions, the tail comparison is:

or

At $p_{20}=0.9$, break-even is $p_{40}\approx0.788943035$, an absolute drop about $11.105697$ percentage points. This is conditional arithmetic, not proof that wider windows meet that reliability. No-overlap windows save $44.444444\%$ of these modeled tokens but disconnect the tail-to-head path. Cutting depth to fifty with the default window/stride gives four calls and $1{,}186.682353\,\mathrm{ms}$ including overhead, a $55.555556\%$ decrease from $2{,}670.035294\,\mathrm{ms}$; recall below the cut is lost. In these scenarios listwise reads $1.8$ times the document tokens but $1.401124$ times the modeled total tokens of pointwise, which omits its own verdict output. Depth, window size, overlap, output lengths and real decode rates jointly determine actual costs.

Decisions and limits

• Treat window twenty/half overlap and tail-to-head motion as one practical starting schedule, not the only possible ranking algorithm.
• Validate integer window boundaries, output reserve, candidate labels and special-token accounting.
• Measure candidate loss when reducing depth or inserting a cascade; measure window reliability and permutation bias when changing width.
• Log repair/fallback rates, gold survival by hand-off and final ranking metrics, not only output/input rank correlation.
• Twenty serial repeated default passes have $39.000706\,\mathrm s$ modeled compute and $53.400706\,\mathrm s$ including the stipulated round trips, before other overhead. Independent repetitions still need resource capacity if run concurrently.
• A cross-encoder-to-twenty cascade saves modeled LLM work but inherits its upstream omissions; compare it on the same judged queries.

22.6 Instruction-aware reranking

What

Instruction-blind scoring uses $s_\theta(q,d)$. An instruction-aware cross-encoder can instead use $s_\theta(I,q,d)$, implemented by a declared concatenation template rather than numeric addition. It permits document states to depend on instruction tokens. Other architectures can also condition on intent; a metadata filter, query encoder, routing policy or other explicit side information can carry it. A document index need not be rebuilt merely because the scorer reads an instruction.

Why

For a deliberately symmetric impossibility example, fix the same query and two candidate documents under two equally likely instructions, with a different gold document for each. Assume the scorer and tie-breaking see neither the instruction nor any correlated side channel. It assigns fixed logits $a,b$, gap $u=a-b$. The sum of the two one-positive list losses is:

The mean floor is $\ln2\approx0.693147181$ nats. For the same $n$ candidates under $n$ equally likely instructions with distinct golds, fixed softmax probabilities $v_i$ obey $-\sum_i\ln v_i\ge n\ln n$; mean loss is at least $\ln n$. An instruction-independent winner has average intent-correct-at-one at most $1/n$. These are symmetry/missing-information bounds, not universal ceilings for all instruction-blind deployments. With unequal intent frequencies the analogous optimal mean log loss is their entropy and best fixed-winner accuracy is the largest frequency. Candidate filtering, query changes or other intent-correlated inputs can invalidate the fixed-input assumptions. The source describes TART-full training on task instructions and a negative pool mixing random and mined instruction violations, nominally $90\%/10\%$. This is reported recipe context, not a proof that all random negatives carry zero instruction gradient or that ten percent is universally sufficient. On-topic instruction-conflicting examples make intent dependence identifiable in the relevant comparisons; labels, coverage and optimization still determine whether the model learns it.

Failure without it

An instruction-conditioned query embedding can select among document directions already represented by a fixed index, including directions learned during instruction-aware training. It is not generally impossible to support new intent without re-embedding documents. A standard nonnegative bag-of-terms score does not enforce a hard logical exclusion by itself, but retrieval systems can combine filters, negative weights or query operators. Explicit instruction input is not sufficient for learning or ranking quality. A single-intent benchmark cannot isolate sensitivity to conflicting instructions; use matched queries/candidates with differing golds. Random-only negative training does not mathematically force no quality improvement.

Cost, complexity, and worked example

The work example uses a separate $4\times10^6$-chunk corpus, the same BERT proxy, $32$ query and $256$ document positions, depth $100$, and an extra $32$ instruction positions. The corpus size does not multiply this rerank cost once the candidate set is fixed. Intent-success predictions below are conditional statements, not benchmark measurements.

At $L=320$, the linear term is $70{,}400{,}000{,}000$ and attention adds $3{,}774{,}873{,}600$ FLOPs. The added depth-hundred work is $2.281537\,\mathrm{ms}$ at the assumed rate, or $11.679465\%$. If exactly ten percent of equally weighted training examples, not merely negatives, are the balanced contradictory pairs above, an instruction-blind model has at least this contribution:

A loss plateau near that value is not a diagnostic proof: other losses, label noise, normalization and regularization can contribute. Verify instruction-dependent ranking on controlled conflicting examples. The extra instruction work is equivalent to this many additional baseline-length candidates in the same proxy:

Only eleven full additional candidates fit that budget; rounding to twelve exceeds it slightly. At the unrounded dense endpoint slope, increasing depth one hundred to one hundred eleven would change the fitted candidate ceiling by:

This is a hypothetical compute exchange, not an observed equivalence between intent quality and recall. Twelve extra candidates would give about $0.492905$ fitted points but cost more than the exact budget. The two-point length exponent predicts:

The fitted increase is approximately $12.142355\%$, versus $11.679465\%$ from the explicit linear-plus-quadratic proxy. A two-point power-law fit is not identical to that sum, whose local elasticity varies with length. Neither is a measured hardware scaling law.

Decisions and limits

• Prefer reliable metadata constraints for hard requirements when available, without mistaking approximate ranking preference for enforcement.
• Give the scorer a documented intent input and train/test on topical conflicts where the intended winner changes.
• Treat the source's ten-percent mined pool as a recipe to validate, not a universal information threshold.
• Swap instructions on matched queries/candidates and inspect pairwise margins, gold order and success. Absolute score changes alone can reflect a common offset rather than useful rank changes.
• Test instruction robustness across tasks and free-text phrasing; a fixed one-instruction-per-task dataset can have confounds.
• Keep the document encoder/index contract explicit. A changed query-side intent does not automatically require all four million documents to be re-embedded; changing the stored document representation may.

22.7 Document-bound versus model-bound

What

For a fixed first-stage pool, let $h_m=R_1C(m)$ be the probability that specified gold evidence is in context. In general, answer success is $E(m)=A_mh_m+B_m(1-h_m)$, where $A_m$ and $B_m$ are success probabilities with and without that evidence. The following toy sets $B_m=0$ and the same $A_m=A$ for both context sizes and rankers:

Compare $E(m+1)-E(m)$ with $E_{\mathrm{new}}(m)-E(m)$ for a specific available ranker. Calling the larger document gain “document-bound” and the larger ranker gain “model-bound” is an operational label, not an intrinsic pipeline property. If the interventions have different costs, also compare gains per added latency, price or other constrained resource; raw gains alone do not identify the better investment.

Why

Use an unbounded geometric placement surrogate with $0\lt p_{\mathrm{rank}}\lt 1$. A real candidate set has finite size $k$; the next equation is the surrogate, not an exactly normalized finite-pool rank law:

With $q_{\mathrm{rank}}=1-p_{\mathrm{rank}}$, the exact version conditioned on rank at most $k$ and its large-pool approximation are:

The source reports $\mathop{\mathrm{MRR}}\nolimits @10=0.365$ in a different MS MARCO reranking comparison. The chosen geometric parameter $p=0.18$ gives $\sum_{r=1}^{10}p(1-p)^{r-1}/r\approx0.366885810$, not the untruncated mean $-p\ln p/(1-p)\approx0.376419167$. Numerical closeness does not identify the true rank distribution, and conditional-on-retrieval MRR cannot be fitted directly to an unconditional score without accounting for misses. Combining this illustrative parameter with DPR Natural Questions recall is a scenario, not one benchmark fit. In the unbounded surrogate, one more document gains:

Upgrading from $q$ to the specified smaller $q_{\mathrm{new}}$ changes the large-pool conditional coverage by:

Equating the two gives

The common positive factors $A,R_1$ cancel only because this scenario holds them fixed for both interventions. If extraction changes with context order/length, or the first-stage pool changes, they need not cancel. For $p=0.18,q=0.82$ and $p_{\mathrm{new}}=0.212,q_{\mathrm{new}}=0.788$:

At integer $m=4$, document/ranker coverage gains are $8.138192$ and $6.655031$ percentage points; at $m=6$, they are $5.472120$ and $6.458839$. Thus the surrogate favors another document through $m=4$ and the specific ranker from $m=5$. At five, the end-to-end gain difference is only about $0.011296$ percentage points after the common scale, far below many practical measurement resolutions. The $0.032$ increase is in the geometric parameter, not an observed $3.2$-point MRR or answer-quality gain. Finite-pool normalizers depend on each ranker and alter the exact crossing; at $k=100$ their corrections are tiny here, but the closed-form formula is still the unbounded approximation.

Failure without it

A removal ablation tests the total contribution of the current reranker, not the marginal return from an available upgrade. Increasing context coverage is monotone in the placement toy; real answer success need not be, because extra distractors, positional changes and extraction behavior can hurt. There is no general theorem that real answer quality follows a particular U-shaped curve or that a single context-count threshold captures every harm. Measure the relevant interventions and budget at the current operating point.

Cost, complexity, and worked example

For the illustrative combined scenario take $R_1=0.854,A=0.75,p=0.18,p_{\mathrm{new}}=0.212$. The following table uses the large-pool geometric approximation, with common factor: