Chapter 32: Evaluating Retrieval

00-START-HERE · 01-READING-ROUTES · 31_attribution_and_citation · 33_evaluating_generation Purpose: evaluate retrieval coverage and order at the evidence actually consumed, distinguish judged-pool metrics from full-corpus quantities, and validate scalable relevance judges without confusing illustrative costs or macro-F1 with deployment guarantees.

TL;DR

• At a fixed prefix, precision, recall and set-$F_1$ ignore permutations within that prefix. Changing which documents cross the actual context cutoff can change them.
• Report coverage at the consumed evidence slice and rank quality with explicit cutoffs, grades and query averaging. Token packing and truncation may produce a query-dependent depth rather than one fixed $k$.
• MRR uses only the first qualifying hit. nDCG uses discounted gains through its cutoff and a per-query ideal; neither guarantees that all jointly required evidence is present.
• Missing labels require a judgment strategy, not a mathematical workaround. Judging only a retriever’s candidates does not establish full-corpus recall or a complete ideal ranking.
• Separate prompt exemplars, tuning data and held-out human validation. An LLM judge scales a proxy; human checks, unknown-label reporting and version control remain necessary.
• Balanced, boundary-focused exemplars are a strategy to test, not a guarantee of unbiased judgment or higher macro-$F_1$.
• Inspect confusion counts, per-class recall/precision, macro-$F_1$, prevalence and action-specific harms. Accuracy can hide rare failures, while macro-$F_1$ is neither accuracy nor prevalence-invariant.

The story

Think of retrieval as the stage crew for a play. The corpus is a prop warehouse and the query is the scene request. A relevance checklist states which props count under a specified rubric. The checklist may be incomplete or disputed; a label is not an infallible certificate. The retriever sends an ordered cart toward the stage. Set metrics inspect the cart after ten slots. They ask how many correct props are somewhere on it. They do not ask where those props sit. A cart with every needed prop in slots 1 through 4 looks identical to one with the same props in slots 7 through 10. The generator is the actor. In this simplified scene it can access only the first five slots. The first cart delivers all four needed props; the second delivers none inside that cutoff. This is a hard-access example, not a claim that language models invariably prefer earlier context or successfully use every accessible passage. MRR resembles a stage manager who stops scoring at the first qualifying prop. It does not score later requirements or differences above its binary threshold. nDCG keeps scoring gains through the cutoff, but its sum still need not capture interactions between props that are useful only together. Now remove the checklist. Humans or models can propose relevance judgments, but checking only the arriving cart cannot reveal every useful prop still hidden in the warehouse. Precision over judged results may be available while full-corpus recall remains unknown. A small human anchor can help validate a scalable judge; it does not complete the relevance set by itself. Rehearsal examples can influence the judge. Under an IID $90:10$ class mixture, eight draws have expected counts $7.2$ and $0.8$, and about a $43.05\%$ chance of no minority example. Four examples per class guarantees prompt representation, not correct decisions. Macro-$F_1$ weights class scores equally, but must still be read with the confusion matrix and uncertainty. The evaluation is one pipeline with separate responsibilities: candidate coverage, final ordering and packing, relevance judgments, and judge validation. Use a shared judgment protocol, distinguish unknowns from negatives, and measure action-specific errors. Balanced examples and one aggregate score cannot certify this pipeline.

Decoder table

Core mechanics

Mathematical conventions

Use one declared evaluation unit: a document, passage or deduplicated evidence group. For query $q$, let $U_q$ be the evaluation universe, $R_q\subseteq U_q$ the binary relevant set, and $(d_{q,1},\ldots,d_{q,k})$ a ranked prefix of exactly $k$ distinct units. Let $S_{q,k}$ be its set and $h_q(k)=|S_{q,k}\cap R_q|$. In the basic formulas $1\le k\le|U_q|$; shorter lists require a declared denominator/padding policy. Duplicates must not earn repeated gain merely by occupying more ranks. Recall requires $|R_q|\gt 0$. When no relevant unit is known, distinguish truly unanswerable queries from incomplete judgments; declare an exclusion/zero convention and report their count. For nDCG, fix the grade-to-gain function, logarithmic discount, judgment universe and zero-IDCG policy. For MRR, fix a binary threshold and cutoff. Treat unjudged units as unknown, not proven irrelevant. Averages below give equal weight to each eligible query, not each document pair. Query-weighted and pooled-count metrics differ. Compare systems on the same queries, relevance protocol and source versions; use paired query-level uncertainty analysis when queries are the independent sampling units. Ten thousand pair labels from five hundred queries are not ten thousand independent query outcomes. Keep full precision through all calculations and round only the reported result. Judge agreement, probability calibration, relevance, claim support and answer truth are distinct. Human labels have uncertainty too. Historical results and hand-constructed confusion matrices do not prove a prompted-model ceiling or an exemplar strategy’s causal effect. Cost examples exclude unspecified overhead and require their stated price, token and scheduling assumptions.

32.1 Set metrics vs rank metrics

What it measures

Fix query $q$ and the conventions above. Define $P_q(k)$ and $r_q(k)$ as precision and recall, reserving $R_q$ for the relevant set. The formulas assume exactly $k$ distinct returned units and positive relevant-set size where recall is used:

For a no-hit prefix with $k\gt 0$ and $|R_q|\gt 0$, set $F_{1,q}@k=0$ using the count form; the literal harmonic-mean expression is $0/0$. Recall is undefined when $|R_q|=0$ unless an explicit reporting convention is imposed. A truly empty prediction list also has undefined ordinary precision; that is not the same situation as a nonempty prefix with zero hits. With fewer than $k$ results, fixed-$k$ precision pads missing positions as nonrelevant, whereas returned-list precision divides by the actual returned count. Name the convention. Query averaging also matters. One query with one of one relevant item recovered and another with zero of one hundred gives mean query recall $0.5$, but pooled-count recall $1/101\approx0.009901$. Do not change the weighting silently. Plain accuracy is:

As a deliberately misleading corpus-wide binary classification metric, mark every unretrieved document negative. With $N=10^7$ and four relevant documents, retrieving nothing gives $\mathrm{TP}=\mathrm{FP}=0$, $\mathrm{FN}=4$ and $\mathrm{TN}=10^7-4$:

The decimal is exact for these integer counts. The original source’s $99.9996\%$ is not the same value. This all-negative classifier has zero recall, and empty-list precision needs its own convention; high corpus accuracy does not imply useful retrieval. True negatives dominate this corpus-wide classification construction. Accuracy can still be informative for a relevance judge on a declared labeled population, but it must be interpreted with prevalence and class errors rather than treated as retrieval success. At a fixed cutoff, these set metrics depend only on $h_q(k)$, $k$ and $|R_q|$. Permuting the same prefix preserves them. Moving a relevant document across the cutoff changes the prefix and can change the metrics; “order-blind” does not mean cutoff-independent. Rank metrics score positions within a declared cutoff. Reciprocal rank awards $1/r$ only to the first qualifying hit. DCG sums gain times $1/\log_2(i+1)$ through rank $k$. Swapping equal-gain items, or changing ranks after a metric has stopped scoring, need not change a rank-aware metric.

Why it matters

A production prompt is limited by tokens and serialization, not necessarily a fixed number of documents. Measure the final packed evidence after filtering, deduplication and truncation; record partial chunks and per-query depth. Relevance does not guarantee successful reader use, and position effects need not decrease monotonically from the start of context. The worked example fixes four relevant chunks and a ten-document result set. System A puts the four hits at ranks 1 through 4. System B puts the same four hits at ranks 7 through 10. At $k = 10$, both systems score:

For this toy, use a $4{,}096$-token total budget and exactly five complete $500$-token chunks: $2{,}500$ chunk tokens leave $1{,}596$ for the question, instructions, delimiters and any reserved output. Whether that is feasible depends on the actual tokenizer and request layout; it is not a measured universal “4k prompt” configuration. At $k=5$, System A has precision $4/5=0.8$ and recall one; System B has both zero. At depth ten, their first-hit reciprocal ranks are:

The chapter’s historical MS MARCO passage-ranking example reports MRR@10, illustrating a first-hit objective rather than set accuracy. Benchmark relevance definitions, pooling and official evaluator versions are part of that result; it is not a universal objective for every RAG task.

Failure without it

A dashboard using only set metrics at depth ten calls these two orders a tie, while their set metrics at the actual five-item cutoff already distinguish them. Rank metrics add information within a fixed prefix; the wrong cutoff is a separate measurement error. An accuracy dashboard can reward a retriever that returns nothing. A pure reranker that only selects/reorders candidates cannot recover an absent unit. If its output set $T_{q,k}\subseteq C_{q,d}$ with $k\le d$, then $|T_{q,k}\cap R_q|\le|C_{q,d}\cap R_q|$. Thus candidate recall bounds final recall under the same unit and relevance definition. Query expansion, additional search or generated evidence changes that assumption. A single $F_1@k$ number therefore cannot establish that a retriever change helped downstream use.

Cost and decision rules

Once prefix labels, relevant-set size and IDCG are available, hit counts, first hit and DCG can be accumulated in one $O(k)$ prefix pass. Obtaining valid labels and the ideal normalizer may cost much more. A shared candidate pool improves consistency, but a pool missing relevant units does not supply a full-corpus denominator.

• Evaluate the evidence actually consumed, including token packing and per-query cutoffs. Curves over depth or token budget remain useful for diagnosis even after an operating point is fixed.
• Pair coverage with an appropriate rank metric and a downstream task check. A rank-aware automated judge still requires validation and an explicit target.
• Do not use all-corpus accuracy to approve retrieval. For judge accuracy, disclose the evaluated class mixture, class errors and costs; a deliberately balanced test is not the only permissible population.
• Compare systems against shared, versioned judgments and a sufficiently broad pool. Labeling each system’s own output separately can create different denominators and selection bias.
• Measure candidate recall and final-prefix quality separately, even in a simple pipeline. Report no-answer cases, unjudged results, uncertainty and the limits of the labeled universe.

32.2 MRR, nDCG, and $\mathrm{recall}@k$ derived

What MRR measures

For each query, let $\rho_q$ be the first rank whose grade meets the declared binary threshold, and let $\rho_q=\infty$ when there is no such unit. The recall function remains $r_q(k)$. The cutoff version is:

For a nonempty eligible query set $Q$, MRR@k averages one term per query, including zero-hit queries:

MRR stops scoring after the first qualifying document. It ignores later hits and distinctions among grades that meet the same threshold. It also gives no credit for a hit below its cutoff. Do not average only over successful queries; that changes the metric. For one query with hits at ranks one and ten, $(1+1/10)/10=0.11$ is not MRR; the reciprocal-rank term is one. Nor is an arbitrary sum of all relevant-hit reciprocals automatically Reciprocal Rank Fusion. RRF assigns each candidate $d$ a score such as $\sum_\ell1/(\kappa+\mathop{\mathrm{rank}}\nolimits_\ell(d))$ across input lists containing $d$, with a declared offset $\kappa$. It is a different, per-candidate fusion operation.

What nDCG measures

Use illustrative grades $\mathrm{rel}_{q,i}\in\{0,1,2,3\}$ under an explicit rubric. Fix a nonnegative gain function $g$ with $g(0)=0$. This chapter’s worked numbers use linear gain $g(r)=r$; another common choice is $g(r)=2^r-1$. Different gains change values and potentially system comparisons; use the same choice in DCG and IDCG. Cumulative gain sums the grades:

CG uses grades but ignores permutations of the same prefix. DCG adds the logarithmic rank discount:

The plus one keeps rank 1 undiscounted:

The discount weight is $1/\log_2(i+1)$, not the denominator itself. It is one at rank one; the denominators at ranks two and ten are about $1.584963$ and $3.459432$. For integer ranks it decays more slowly than $1/i$ after rank one. This is a chosen utility model, not a measurement that a generator uses evidence according to that exact law. Let $\mathrm{rel}^{\star}_{q,1},\ldots$ be grades sorted in descending order over the declared evaluation universe, not merely the current retriever’s candidates. Using the same gain and cutoff gives

If there are fewer than $k$ units, pad the ideal list with zero gains under the declared policy. Incomplete qrels supply an ideal over known judgments, not a certified corpus-wide optimum. Dividing by each system’s own candidate-only ideal can hide retrieval failures and make comparisons inconsistent.

With nonnegative gains, the same discount and an ideal covering all evaluated units, the score lies in $[0,1]$. A value of one means optimal gain through this cutoff, possibly under many tied orders; it says nothing about later ranks or full recall. When IDCG is zero, the ratio is undefined. Choose and report a zero/exclusion convention and the affected query count, rather than silently dropping failures or asserting every query is normalizable.

Why the three metrics disagree

There are exactly two positive-grade documents in the declared universe: A has grade three and B grade one. All others have grade zero. Use $k=10$, linear gain and, initially, binary threshold $\mathrm{rel}\ge1$ for recall and MRR. Configuration X puts B at rank 1 and A at rank 4. Configuration Y puts A at rank 1 and B at rank 4. At threshold one, both have RR@10 equal to one and recall@10 equal to one. For a one-query set, the MRR values also tie. If relevance instead requires grade at least two, X’s first qualifying hit is at rank four and Y’s at rank one, giving $0.25$ versus one. The claimed tie depends on the threshold. For Configuration X:

For Configuration Y:

The ideal order puts A at rank 1 and B at rank 2:

The normalized scores are:

The difference, calculated before rounding, is approximately $0.313596506$ nDCG units. These are ratio scores, not probabilities of a correct answer. The displayed three-decimal summaries are $0.631$ and $0.945$, with gap about $0.314$; do not use rounded DCG/IDCG as if they were exact inputs. The chapter uses TREC Deep Learning’s graded judgments and nDCG@10 as a historical benchmark example. Reproduce a benchmark with its official grade mapping, gain function, unjudged policy and evaluator; the linear-gain worked toy is not a claim that every official scorer uses this exact convention.

Failure without it

MRR can equal one when the first hit is present but two of three jointly required supporting chunks are absent. Per-item relevance metrics do not fully describe evidence complementarity, redundancy or whether a complete support set survives packing. Add a task-specific all-required-evidence or answerability check where needed. With graded data, MRR also ignores how strong its first qualifying hit is. That limitation depends on the chosen threshold; it is not correct to claim that every positive-grade hit must count under every benchmark’s binary relevance rule. CG has the same order blindness as a set sum. Raw DCG depends on available gains and grades, as well as rank. Per-query normalization supports a declared cross-query aggregation but does not make tasks identical. Report which eligible queries are averaged and whether the aggregate is equally query-weighted. nDCG does penalize missing high-gain items when the common ideal includes them; it is not purely ordering within the returned set. But nDCG alone does not establish full recall. With three equal relevant items and $k=1$, a relevant first hit gives nDCG@1 equal to one and recall@1 equal to $1/3$. Separately, if only a grade-one candidate is retrieved while a grade-three source is absent, candidate-only nDCG@1 is one but full-universe linear-gain nDCG@1 is $1/3$. These are different denominators.

Cost and decision rules

With prefix grades and a precomputed normalizer, MRR and DCG are linear-time prefix reductions. Computing IDCG requires knowing the relevant grades over the chosen universe and selecting/sorting their best prefix; it is not obtained just by reordering one system’s results. The harder cost may be judgment completeness and agreement, not summing the formulas.

• Choose rank and coverage metrics from the task. nDCG handles graded utility but does not by itself guarantee complete multi-hop evidence; MRR is suitable for a declared first-hit objective.
• Fix binary thresholds, gain functions, discount, cutoff, duplicate handling and query weighting before comparing releases. Report any change in these definitions.
• Score the actual consumed slice and useful depth/token-budget curves. A good deep score need not imply a good shallow prefix.
• Measure both candidate-depth coverage and final-prefix coverage where both matter; candidate recall is not final recall.
• Use shared per-query IDCG over a declared judgment universe. Do not normalize by a global constant or by each system’s own result list. Report incomplete/unjudged cases and zero-IDCG handling.

32.3 Evaluating without ground truth

What changes when $R$ is missing

A metric needs the labels it actually uses. Precision at a fixed prefix needs that prefix’s binary labels; MRR@k needs qualifying-hit labels through $k$. Full-corpus recall additionally needs the total relevant-set size. nDCG needs a graded ideal over a declared universe. Without adequate judgments these target quantities cannot be evaluated exactly, although partial, pooled or estimated metrics can be reported with their limits. One of three sources must supply labels:

1. Humans label query-document pairs using a rubric, with adjudication and uncertainty where needed.
2. A prompted LLM proposes labels or grades, checked against independent human judgments.
3. A trained classifier supplies a validated proxy, with training-data and distribution-shift limitations. Implicit feedback or other weak labels are additional proxies, not automatic replacements for relevance judgments. Binary labels are insufficient to reproduce an arbitrary graded nDCG target. Humans provide task-specific reference judgments, not error-free truth. Expertise, ambiguity, source access, time budget and inter-rater disagreement matter. Reusing a stable judgment pool can avoid relabeling unchanged pairs every release, but new results, changed content and rubric changes require maintenance. A zero-shot judge can receive a fully explicit rubric without worked examples; zero-shot does not necessarily mean an unstated standard. Without validation, however, its application of that rubric remains unverified. Mere entity mention may be inadequate when the task requires answering the query or supplying a particular kind of evidence. A small human set helps develop the rubric and select examples, but does not automatically fix judge error. Separate prompt exemplars and tuning cases from locked held-out validation; shared queries, documents and near-duplicates can leak information across splits. Fresh human checks measure drift and disagreements, with enough minority/graded cases for the intended claim. CRAG fine-tunes a T5-large retrieval evaluator to score each question-document pair. The original paper uses positive/negative training targets $1$ and $-1$, with inference scores in $[-1,1]$, then upper/lower thresholds to trigger Correct, Incorrect or Ambiguous actions. At list level, a high-scoring document can trigger Correct; all scores below the lower threshold trigger Incorrect; other cases are Ambiguous. This is not an embedding cosine threshold or simply a direct three-class gold-label predictor. Its score is not automatically a calibrated probability.[1]

Why calibration matters

A validated judge can amortize human supervision, while remaining a fallible proxy. Check relevance decisions at the intended granularity, evidence scope, domain and class mixture; a good claim-support judge is not automatically a good query-document relevance judge. The default pipeline has four steps:

1. Define the unit, relevance rubric, grade scale, evidence scope and treatment of unknowns.
2. Label a starting human set—roughly one hundred pairs is a workflow suggestion, not a statistical guarantee—and separate development from held-out assessment.
3. Select a bounded exemplar set from development data; fix the judge model, prompt, parsing and grade mapping.
4. Judge a shared, appropriately broad candidate pool and validate on held-out/fresh human cases. Record error types, uncertainty, unjudged coverage and version changes. The judge’s positive labels form a pseudo-relevant set only within the assessed universe. If it sees only the top twenty returned items and labels one relevant, dividing that hit by that one known positive gives pool recall one. The same observed labels are compatible with nine additional relevant corpus items never judged, in which case corpus recall is $1/10$. More labels on the same selected cart do not identify the unseen denominator. Pool across diverse systems, add targeted or probability-sampled assessment where appropriate, and report any completeness assumptions. Sampling weights and uncertainty are needed for design-based corpus estimates. The chapter names Haystack’s evaluation tooling as one possible harness. Tool choice does not establish human agreement, complete qrels, unbiased pooling or a calibrated probability model; those are evaluation-design responsibilities.

Worked cost example

For a one-system illustrative pass, take $500$ held-out queries and exactly twenty distinct candidates per query. This produces pair-judgment occurrences, not a complete set of relevance judgments for the corpus:

For a comparison, judge a shared union of candidates; its size need not remain ten thousand. Deduplication and cached labels may reduce calls, while adjudication and changed content can add work. The following budget assumes ten thousand fresh, one-call pairs each pass. For one human judgment per pair, assume two judgments per minute, or $120$ per rater-hour. Training raters, agreement checks and adjudication are excluded:

At a hypothetical $\$25$ per rater-hour, retaining full precision:

One rater requires approximately $10.416667$ eight-hour workdays. Ten equally productive parallel raters require about $8.333333$ hours of ideal elapsed labeling time, not $8.333333$ total rater-hours. Coordination and adjudication add time. In a separate short-pair LLM toy, assume $300$ total input tokens—including the query, passage, instructions, delimiters and chosen exemplars—and twenty output tokens per call. This is not the earlier $500$-token-chunk prompt. Prices are hypothetical: $\$0.50$ per million input tokens and $\$1.50$ per million output tokens. If the actual few-shot layout does not fit, increase the input budget rather than omit it from the bill.

Ten thousand calls process $3.0\times10^6$ input and $200{,}000$ output tokens. Their generation bill is:

Assume fixed one-second service per call, fifty available slots, adequate rate/token limits and negligible overhead. Then ideal batched elapsed service is:

Fifty calls per second requires about $15{,}000$ input and $1{,}000$ output tokens per second under this layout; concurrency alone does not supply that throughput. A human sample of one hundred pairs at the same pace takes fifty minutes, or $5/6$ rater-hour, costing $\$125/6\approx\$20.833333$. Split its development and held-out uses appropriately; its size does not certify judge quality. Across ten fresh passes, the human-only labeling bill is $10(\$2{,}083.333333\ldots)\approx\$20{,}833.33$, with $833.333333$ rater-hours. This assumes unchanged throughput, ten thousand new judgments each pass and no label reuse. For the LLM path, explicitly collect one hundred human labels before pass one and refresh with another hundred before passes four, seven and ten: four samples total. Keep sample uses independent of held-out scoring. The ten-pass bill is

Using the rounded $\$21$ sample price gives $\$102$, but that is not the full-precision result. Machine service totals $2{,}000\,\mathrm s\approx33.333333$ minutes. The four samples add $200$ human minutes; with one rater and serial service, combined work is $233.333333$ minutes, or about $3.888889$ hours. “Under one hour” describes machine service alone, not this human-inclusive serial workflow. Parallel human work or pre-existing labels can change elapsed time, but not eliminate labor. The one-pass human/LLM token-bill ratio is about $1{,}157.407407$, but the ten-pass ratio including these four human samples is only $205.592105$. Ideal one-pass elapsed ratios are $1{,}500$ with one human rater and $150$ with ten raters; neither is a universal latency claim. Thus “three orders of magnitude in total cost and latency” does not describe every ledger above. If $s$ extra input tokens per call were omitted from the $300$-token layout, the same price adds $\$0.005s$ per ten-thousand-call pass. For $s=800$, generation rises from $\$1.80$ to $\$5.80$. All budgets still omit rubric development, expert adjudication, judge selection, retries, infrastructure and resulting error costs, and do not establish equal label quality. Pooling diverse top-ranked results is a practical benchmark strategy, not proof of exhaustive relevance coverage. New systems can find relevant unjudged items outside an older pool. Maintain and audit qrels, and distinguish an unjudged-as-zero scoring convention from a verified negative label.

Failure without it

A judge with no held-out agreement and error analysis produces an unvalidated proxy, whether zero-shot or few-shot. Agreement alone can also be high when both judge and reference share a bias or one class dominates. Judge model/prompt changes and traffic shifts can change scores even if retrieval is unchanged. Pin and log versions, retain comparable labeled cases, and use fresh checks; no fixed recalibration interval guarantees stability. Retraining a retriever against disputed or adaptively optimized judge labels can fit proxy errors. Keep independent human evaluation and avoid reusing a public tuning pool as the sole release test. A trained classifier can fail after a domain shift even when it was reliable on its training distribution.

Cost and decision rules

• Start with an explicit rubric and independent validation. One hundred labels is a planning suggestion, not a prevalence-free accuracy guarantee; rare classes or many domains can need much more.
• Separate exemplar selection, prompt/model tuning and final assessment. Retain representative easy cases as well as useful boundary cases; audit grade agreement for graded metrics.
• Validate after judge changes and monitor drift using risk-appropriate fresh samples. A pinned model does not freeze the query/source distribution.
• Compare prompted and trained judges on actual latency, total cost, error types and domain transfer. Neither architecture is restricted to one mandatory serving/evaluation role.
• Use human adjudication where stakes or disagreement warrant it. Reuse sound stable labels, expand pools for new results, and do not call top-candidate pseudo-recall full-corpus recall.
• Set gates from uncertainty and action-specific harm, not raw agreement alone. If a gate fails, diagnose rubric, evidence, prompt or model problems before trusting proxy improvements.

32.4 Judge design, class balance, and macro-$F_1$

What accuracy hides

The following claim-support judge is a separate illustrative task: it labels a generated claim against its cited passage. Query-document retrieval relevance can require a different rubric or graded labels; transferring this judge is a hypothesis to validate, not an identity between tasks. On a $90\%$-attributable population, an always-attributable judge has exactly $90\%$ accuracy and zero recall for non-attributable cases. With the two classes fixed and zero unsupported-class F1, its macro-$F_1$ is $9/19\approx0.473684$, not $0.90$. A mostly-positive judge need not have those exact counts, so inspect its actual confusion matrix. Accuracy pools both classes:

For each class $c$ in a declared fixed class set, compute one-versus-rest counts. Precision and recall require positive denominators; declare a convention when no cases are predicted or present. The count form of F1 avoids a spurious $0/0$ when both computed precision and recall are zero:

Then it gives every class equal weight:

With two fixed classes, the class represented by twenty cases counts equally in the macro average to the one with one hundred eighty. But class precision depends on prevalence, so macro-$F_1$ is not invariant to the evaluation base rate. Identical $0.90$ class recalls give macro-$F_1=0.90$ on balanced labels but about $0.792359$ at a $90:10$ mixture. Report per-class counts and the actual evaluation population, not just an equal-weight average. AttributionBench reports a historical fine-tuned GPT-3.5 result near $80\%$ binary macro-$F_1$. That is not an error probability of one in five, a present-day ceiling or a mathematical limit for few-shot prompts. For example, counts $(\mathrm{TP},\mathrm{FN},\mathrm{FP},\mathrm{TN})=(27,3,3,7)$ give macro-$F_1=0.80$ but accuracy $0.85$. A prompted result above a published reference requires a fair independent evaluation, not automatic distrust solely because it exceeds that number.[2]

Why exemplar balance matters

Prompt construction chooses the exemplar distribution; it does not choose the class balance of independently sampled evaluation traffic. Prompt examples may shift the judge’s label prior or threshold, but a behavioral effect must be measured rather than assumed. For eight IID draws with minority probability $0.1$, let $X\sim\mathop{\mathrm{Binomial}}\nolimits(8,0.1)$. Then $\mathbb E[X]=0.8$ and the majority expectation is $7.2$, not a guaranteed seven-to-one draw. Moreover,

Sampling without replacement from exactly one hundred eighty majority and twenty minority examples is hypergeometric instead, with zero-minority probability $\binom{180}{8}/\binom{200}{8}\approx0.423644$. Do not substitute an IID probability as an exact finite-population result. Selecting four exemplars per class guarantees representation in that eight-example prompt. Boundary cases can reveal distinctions, while representative easy cases can anchor the rubric. Neither balancing nor hard examples guarantees better classification, removes every shortcut or calibrates production probabilities. Validate selection and ordering on independent data, and preserve the intended evaluation base rate. The original AttributionBench error audit examined $321$ selected GPT-3.5 chain-of-thought prompting errors, not three hundred errors from the separately fine-tuned system. It reports $66.0\%$ fine-grained information insensitivity, including missed nuance and inference problems. The selection uses up to fifty errors per dataset and class balancing for several sets; these are not unbiased production error proportions.[2] A further $26.8\%$ is information-access mismatch, including evidence/context differences and possible label issues. If support is scored only against the generator’s snippet, adding unseen full-page evidence changes the target rather than proving that the snippet supported the claim. Align authorized evidence and annotation scope, repair labels where justified, and measure whether an intervention actually reduces errors; access parity does not guarantee a cure.[2] The benchmark uses four in-distribution and three held-out out-of-distribution datasets. This tests particular transfers; success does not prove that a system learned a general rule rather than shortcuts, or cover every future shift.[2]

Worked class-balance example

Use the same $200$ held-out triples for both hand-constructed configurations: $180$ attributable and twenty non-attributable. Rows are true labels and columns predictions, each ordered as attributable $A$, then non-attributable $\neg A$:

These are stipulated outcome counts, not measured results or predictions derived from the exemplar ratios. Their values illustrate metric behavior. They do not establish that stratifying examples caused the difference or that a published macro score bounds either configuration. Configuration 1 uses seven attributable exemplars and one non-attributable exemplar. The judge correctly calls 171 of 180 attributable pairs. It correctly calls 6 of 20 non-attributable pairs. The recalls are $0.95$ and $0.30$. It predicts 185 cases as attributable. That total is 171 correct calls plus 14 missed non-attributable cases. It predicts 15 as non-attributable. That total is 6 correct calls plus 9 missed attributable cases.

The $88.5\%$ accuracy coexists with only $30\%$ recall for unsupported cases: fourteen of twenty broken citations are falsely accepted. The macro average exposes a class problem but does not by itself quantify its harm or set a safe action threshold. Configuration 2 uses four exemplars from each class. Configuration two correctly labels $165$ of $180$ attributable cases and fifteen of twenty non-attributable ones. Recalls are exactly $11/12\approx0.916667$ and $3/4=0.75$. It predicts 170 cases as attributable. That total is 165 correct calls plus 5 missed non-attributable cases. It predicts 30 as non-attributable. That total is 15 correct calls plus 15 missed attributable cases.

Computed before rounding, the macro change is approximately $0.131506849$ score units, or $13.150685$ points on a percent-scaled score. The three-decimal summaries are $0.640$ and $0.771$, not $0.772$. Accuracy increases $1.5$ percentage points, from $88.5\%$ to $90\%$. Majority recall falls about $3.333333$ points while minority recall rises $45$ points; false accepts fall from fourteen to five, while false rejections rise from nine to fifteen. Twenty minority cases leave substantial sampling uncertainty. Under an IID within-class Bernoulli illustration, nominal $95\%$ Wilson intervals for recall are about $[0.145477,0.518973]$ for six of twenty and $[0.531299,0.888138]$ for fifteen of twenty. These are separate recall intervals, not an interval or paired significance test for the change. Aggregate confusion matrices do not identify which individual predictions changed. Use paired item-level or query-clustered comparisons, depending on sampling, and account for prompt selection on development data. A historical $0.80$ result is neither a ceiling nor a substitute for this analysis.

Failure without it

Accuracy can approve a judge that misses every minority-class case. Proportional prompt sampling can underrepresent a rare class; balanced sampling can shift other behavior too. Evaluate the resulting confusion matrix and production harm rather than assuming that one prompt ratio must win. Boundary examples can teach useful distinctions but may be ambiguous or unrepresentative. Include clear anchors, adjudicate uncertain labels, and keep the release test separate from the cases used to select prompts. The interview scenario stipulates a macro-$F_1$ drop from $78\%$ in distribution to $61\%$ on a new domain while new-domain accuracy is $89\%$. The macro difference is seventeen score points, not a $17\%$ relative loss. Because the domains and possibly their prevalence differ, inspect per-class rates, samples and costs; the numbers alone do not identify the cause or a universal shipping rule. If necessary evidence is absent, examples alone cannot reveal its specific missing contents reliably. But access mismatch can also involve ambiguity, scope or labels; a fuller input is not a guaranteed fix. Decide whether the intended target is snippet faithfulness or broader external verification before changing evidence.

Cost and decision rules

The attribution-judge example supplies no end-to-end serving bill. Balanced selection, adjudication, prompt-token length, validation and error costs belong in its budget; one judge’s macro score is not a production correctness or risk estimate.

• Report class counts, precision/recall, macro-$F_1$, accuracy, prevalence and uncertainty. Choose release gates from action-specific false accepts/rejections and harm, not a universally sufficient headline score.
• Test balanced and boundary-focused exemplars while retaining representative anchors. Trained models can use weighting or resampling, but these are alternatives to validate, not guaranteed replacements for good examples or data.
• Hold out prompt/model selection from the release test. Do not reject a sound independent result merely because it exceeds a historical reference score.
• Validate relevant domain and traffic slices and monitor future shift, even for a presently narrow domain. No finite OOD set proves general robustness.
• A macro score in the sixties can motivate investigation, not an automatic diagnosis that the prompt alone is wrong. Check evidence, rubric, human disagreement, domain shift and minority errors before deciding the remedy.

Diagrams

Figure 32.1

Figure $32.1$: The same four relevant units occur in each ten-item list. Set metrics at ten tie, but recall at the consumed five-item prefix is one for A and zero for B. RR@10 is one versus $1/7$; RR@5 is one versus zero. This is a fixed-size hard-access toy, not a general token-packing or reader-use guarantee.

Figure 32.2

Figure $32.2$: For configuration X, binary threshold one makes RR@10 equal to one; it ignores the grade-three hit at rank four. Linear-gain DCG includes that hit. Its value about $2.292030$ is DCG, not nDCG; division by the shared ideal gives nDCG about $0.631251$. At threshold two, RR@10 instead equals $1/4$.

Figure 32.3

Diagram description and source

flowchart TB
    A["Human rubric and development labels"] --> B["Choose bounded exemplars; pin judge and prompt"]
    B --> C["Judge a shared pool; unjudged corpus remains unknown"]
    C --> D["Scoped proxy metrics, not automatic full-corpus recall"]
    H["Separate held-out human labels; grades and minority cases"] --> V["Validate errors, uncertainty and drift"]
    C --> V
    V -. "Fresh checks; tune only on development data" .-> A

Figure $32.3$: Human development labels and held-out validation have different roles. Automated labels scale the assessed pool, not its completeness. The starting one-hundred-pair budget is a workflow illustration, not a guarantee of valid graded metrics or a statistically adequate rare-class audit.

Figure 32.4

Diagram description and source

flowchart LR
    A1["Evaluation population: 180 attributable, 20 non-attributable"] --> A2["Example prompt: 7 majority and 1 minority"]
    A2 --> A3["Stipulated counts: 171, 9, 14, 6"]
    A3 --> A4["Macro-F1 0.639922; minority recall 0.30"]
    B1["Same evaluation population; do not rebalance test traffic silently"] --> B2["Example prompt: 4 per class"]
    B2 --> B3["Stipulated counts: 165, 15, 5, 15"]
    B3 --> B4["Macro-F1 0.771429; minority recall 0.75"]
    A4 --> V["Illustrative comparison, not a causal experiment or ceiling"]
    B4 --> V

Figure $32.4$: The constructed matrices illustrate how class errors affect macro-$F_1$ and accuracy. They are not predictions deduced from prompt balance. Eight IID draws at a $90:10$ rate have expectations $7.2$ and $0.8$; a specific seven-to-one prompt is one possible draw. Four-per-class selection guarantees representation, not higher held-out quality. Compute from exact counts before rounding; configuration two rounds to $0.771$.

Whiteboard pack

Drawing order

1. Declare the query universe, relevance unit, unknown labels and binary/graded rubric.
2. Draw a candidate list and the actual token-packed slice, with fixed $k$ only for the toy.
3. Write set metrics and their zero/short-list conventions; distinguish query means from pooled counts.
4. Show four hits at ranks one to four versus seven to ten, then score both at ten and five.
5. Define RR@k, MRR@k, gain, DCG and a shared per-query IDCG, including zero-IDCG handling.
6. Show how candidate-only normalization can conceal missing evidence and how perfect shallow nDCG can coexist with low recall.
7. Separate human development examples from held-out judge validation; leave unjudged corpus items unknown.
8. Write both exact confusion matrices, minority errors and macro-$F_1$, without claiming prompt balance caused the difference.
9. Finish with machine-token cost, human calibration labor and uncertainty as separate ledger lines.

Spoken script

First declare relevance and the evidence actually consumed. Set metrics describe overlap at a cutoff; rank metrics describe selected positional utility. MRR stops at the first qualifying hit. nDCG sums gains and divides by a shared per-query ideal, but does not certify complete support. Missing judgments remain unknown: labeling only retrieved candidates cannot identify corpus recall. A human-validated judge can scale a pooled proxy, with separate prompt-development and held-out cases. Check class errors, uncertainty and action costs rather than accuracy or macro-F1 alone. Balanced exemplars are worth testing, not a guarantee. Include human calibration labor before claiming large cost or latency savings.

Interview traps

1. Two retrievers have identical $\mathrm{precision}@10$ and $\mathrm{recall}@10$. Which one ships?

The two depth-ten set numbers do not decide. At ten, both have precision $0.4$, recall one and $F_1=4/7\approx0.571429$. At the consumed five-item prefix, A has recall one and B zero; their RR@5 values are one and zero. RR@10 is one versus $1/7$. Check the final token-packed evidence, rank quality, downstream task success and uncertainty before shipping; a fixed-prefix toy does not decide every production trade-off.

2. Derive recall, MRR, and nDCG instead of quoting them.

Recall divides prefix hits by the total relevant count in a declared universe, when positive. RR@k is the reciprocal first qualifying rank if within $k$, otherwise zero; MRR averages over all eligible queries including failures. DCG sums a fixed nonnegative gain over $\log_2(i+1)$, and nDCG divides by the same gain’s per-query ideal when positive. Declare thresholds, gains, unknowns, zero-IDCG and query weighting. Perfect nDCG through a short cutoff does not prove complete or joint evidence coverage.

3. There is no ground-truth relevant set. Can retrieval still be evaluated?

Yes, but specify what can be measured. Human or automated judgments can label a pool; held-out human validation is needed before treating model labels as a useful proxy. Top-result labels alone do not establish the full recall denominator or corpus-wide ideal. In the separate ten-thousand-call cost toy, one human pass is about $83.333333$ rater-hours and $\$2{,}083.33$; the LLM token bill is $\$1.80$ and ideal service $200$ seconds. Across ten passes plus four human samples, the LLM path is about $\$101.33$ and $3.888889$ hours of serial combined work—not an all-in under-one-hour guarantee or matched label-quality result.

4. A judge reaches $91\%$ accuracy on a $90:10$ gold set. Do you ship it?

Do not ship from accuracy alone. A majority-only judge already earns $90\%$ accuracy. On a $200$-case population with $180$ positives, $91\%$ accuracy could mean all $180$ positives correct but only two of twenty negatives detected: minority recall $10\%$. Request the confusion matrix, class metrics, independent validation and action-specific risk. The constructed prompt comparison gives macro-$F_1\approx0.639922$ versus $0.771429$, not proof of a causal benefit or a universal ceiling. Missing evidence or changed scope needs diagnosis, not merely more examples.

5. The reranking team says it can fix retrieval order. Does retrieval-stage evaluation still matter?

Yes. For a pure subset/reordering stage, final relevant hits cannot exceed candidate relevant hits under the same unit/rubric. Evaluate candidate recall, final consumed-prefix recall and rank quality separately, against common judgments. A reranker-only ideal must not be substituted for a corpus/pooled ideal when claiming end-to-end quality. Additional retrieval changes the candidate boundary; incomplete qrels limit any claimed ceiling.

Key numbers

00-START-HERE · 01-READING-ROUTES · 31_attribution_and_citation · 33_evaluating_generation