Chapter 4: What the LLM Brings and What It Breaks

00-START-HERE · 01-READING-ROUTES · 03_a_repeatable_framework_for_any_rag_design_question · 05_data_privacy_and_the_legal_surface This chapter explains what the Large Language Model (LLM) contributes to Retrieval-Augmented Generation (RAG), what it breaks, and how to assign each failure to the right owner before choosing a fix.

TL;DR

• Pre-training installs capability and uneven factual memory. Supervised fine-tuning shapes answers. Post-training shapes preferences. A datastore owns facts that change.
• Memorized text becomes easier to extract as model size, corpus duplication, and supplied context length rise.
• The model knows common facts better than rare facts. Retrieval helps the rare tail, but an off-target passage can overwrite a correct answer on common facts.
• Faithfulness asks whether a claim follows the supplied context. Factuality asks whether it is true in the world. One score cannot replace the other.
• Factual errors come from outdated data, ambiguous prompts, retention limits, over-generalization, or failed knowledge integration. Retrieval shrinks some causes and enlarges others.
• Verification is statistical because generation varies. It also faces moving truth and shared blind spots between generator and judge.
• Diagnose the stage, error bucket, traffic slice, and verifier oracle before spending money on more training, more retrieval, or a stronger judge.

The story

Picture one newsroom. The reporter is the LLM. Pre-training is the reporter’s broad education from years of reading. It gives writing ability and uneven memory, but it ends at a cutoff date. Supervised fine-tuning is the newsroom stylebook. It teaches the reporter to answer the assignment instead of continuing a random conversation. Post-training is the editor who shapes tone, refusal, and when to say “I do not know.” The datastore is the live news wire. It owns current prices, office holders, policies, provenance, and deletions because editors can update it without retraining the reporter. The archive clerk is the retriever. The clerk is most useful for an obscure local story that the reporter barely saw during training. On a famous story, however, the reporter may already be right, and a bad clipping can make the answer worse. Faithfulness asks whether the reporter copied the supplied clipping correctly. Factuality asks whether the story is actually true today. A stale clipping can produce a faithful but false article. The copy desk sorts each wrong claim by cause. It asks whether the source was stale, the assignment was ambiguous, the reporter forgot a rare fact, the reporter followed a familiar pattern, or several sources were combined badly. The final verifier is another copy editor. If that editor learned the same false rumor as the reporter, greater editing skill does not help. The verifier needs a dated, independent source that the reporter did not already use. The newsroom lesson is simple. Train the reporter for capability and behavior. Use the live wire for changing facts. Retrieve selectively. Grade against the right source. Verify with independent evidence. This passage uses a newsroom to explain what different parts of a retrieval-augmented language model do—and why neither training nor retrieval alone guarantees a correct answer. Its main point: let the model handle language and reasoning, use an updatable datastore for changing facts, retrieve evidence when it helps, and check answers against sources that can independently establish what is true.

• The reporter is the LLM. Pre-training gives it broad language ability and a store of facts picked up from its training material. That memory is uneven, and it may be out of date because training ends at a cutoff.
• The stylebook and editor represent later training. Supervised fine-tuning teaches the model how to answer a task rather than simply continue text. Post-training shapes preferences such as tone, refusal, and when to say “I don’t know.” These stages affect how the model responds; they do not automatically keep its facts current.
• The live news wire is the datastore. It holds facts that may change—such as prices, policies, or office holders—and can be updated or corrected without retraining the model. The retriever is like an archive clerk who finds relevant passages and supplies them to the reporter.
• Retrieval can help or hurt. It is especially useful for obscure facts the model may not know. But if the model already knows a common fact, an irrelevant or incorrect passage can steer it toward a worse answer.
• Faithfulness and factuality are different checks. Faithfulness asks, “Did the answer follow the supplied passage?” Factuality asks, “Is the answer true in the world?” A model can faithfully repeat an outdated passage and still be wrong.
• Errors need different fixes. A stale source calls for updating the datastore; an unclear assignment may need clarification; a forgotten rare fact may need retrieval; a familiar but misleading pattern may need correction; and conflicting passages may need better source handling.
• A verifier also needs good evidence. A second model that shares the first model’s mistaken belief may confirm the error. A dated, independent source is more useful because it gives the verifier evidence beyond what the generator already saw. So the practical lesson is to assign each job to the right component: train for capability and behavior, use the datastore for changing facts, retrieve selectively, and verify against independent evidence.

Decoder table

Core mechanics

Mathematical conventions

Use decimal bytes unless a binary unit is explicit. Forward and training compute use the simplified proxies $2NT$ and $6NT$ at assumed sustained throughput $3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$ and $\$2.50$ per GPU-hour. These are compute-equivalent costs, not measured end-to-end latency, resource reservations or invoices. They omit attention-length effects, bandwidth, communication, search and other overhead. Fractions, percentage points and relative percentage changes are different units. A fitted empirical trend is not a theorem outside its measured range; calculate before rounding.

4.1 Pre-training, SFT, post-training: which stage owns which failure

Stage ownership

The opening assistant was shipped last quarter with a 27-billion-parameter open-weight model fine-tuned on $40{,}000$ internal tickets. It names a vice president who left in March, returns prose when the frontend expects JSON, and invents a refund window for a product line launched six weeks earlier. The stale name, wrong output form, and absent new fact belong to different owners. Pre-training maximizes the next-token objective over collected text:

Here $\theta$ is the full parameter vector. Next-token targets are derived from the text rather than human-written task labels; that does not make curation irrelevant. Pre-training supplies broad capability and uneven factual memory. SFT maximizes $\log P_\theta(Y\mid X)$ over curated input-output examples, shaping instruction following, output format and context use. Preference and verifiable-reward methods further change behavior. The source's view that later stages mainly elicit pre-existing competence is a working hypothesis, not a theorem that SFT or post-training cannot add facts or skills. Test the base checkpoint to diagnose the starting limitation, then measure what the chosen adaptation actually changes. No stage label by itself proves a hard downstream capability ceiling.

What training cannot own

The stale vice president is not a capability, shape, or preference failure. The model learned the fact correctly; then the world changed after the cutoff. Continual pre-training can refresh it, but models with only a few hundred million parameters can suffer substantial catastrophic forgetting. Each refresh then needs replay data that may no longer exist. Model editing changes named associations, but it inherits the enumeration problem of unlearning. You must know every future fact or sequence that will need a change. It also cannot guarantee complete removal and can damage neighboring associations. Scale cannot close the tail economically. Kandpal et al. (2023) extrapolated competitive accuracy on sparsely supported facts to about $10^{18}$ parameters. At two bytes per weight, storage is $2\times10^{18}$ bytes. Dividing by $8\times10^{10}$ bytes per $80\,\mathrm{GB}$ accelerator gives $2.5\times10^{7}$ accelerators before processing one token. The datastore owns facts that outlive gradient descent because it supports freshness, provenance, and deletion. Deleting a datastore row removes future retrieved exposure, but it cannot erase text the base model already memorized during pre-training. Domain SFT loses as a knowledge patch. It raises $P_\theta(Y \mid X)$ for written examples but does not erase far more frequent contradictory pre-training evidence. It creates internal conflict and returns the bill when the fact changes again.

Worked example and costs

The assistant fails $370$ of $1{,}000$ queries: $180$ stale facts, $90$ contradictions of correct context, $60$ format violations and $40$ low-support-entity errors. These are diagnostic buckets, not guaranteed repair yields. A weight refresh with $N=27\times10^9$ and $D=10^{10}$ uses $108\,\mathrm{GB}$ of fp32 weights. A specified $16\,\mathrm{B/parameter}$ Adam layout totals $432\,\mathrm{GB}$ before activations, an aggregate six-card minimum at $80\,\mathrm{GB}$ each, not six cards reserved solely for optimizer moments. Training work is approximately $1.62\times10^{21}\,\mathrm{FLOP}$, equivalent to $4.76471\times10^6\,\mathrm{s}$, $1{,}323.53$ GPU-hours and $\$3{,}308.82$ at the stated rate. Whether it fixes any of the stale or tail answers must be tested. Five $400$-token chunks add $2{,}000$ prefill tokens: approximately $1.08\times10^{14}\,\mathrm{FLOP}$, $0.31765$ compute-equivalent GPU-seconds or $\$2.20588\times10^{-4}$ per query. One refresh bill equals $15{,}000{,}000$ such incremental prefill components—not that many complete RAG requests. Retrieval still needs search, embedding, source maintenance and answer decoding. For behavior repair, supervised examples can target format and context-use failures but do not guarantee their elimination. Rank-$16$ factors for a $4{,}096\times4{,}096$ update store $16(4096+4096)=131{,}072$ values instead of $16{,}777{,}216$, a $128$-fold update-storage reduction. That is not a $128$-fold reduction in total model memory or training work, nor proof that the update changes no factual answer. The $405$B, $15.6$T cross-check gives $6ND=3.7908\times10^{25}\,\mathrm{FLOP}$, approximately $0.242\%$ below the rounded reported $3.8\times10^{25}$. Dividing the proxy by $30.84$ million GPU-hours implies approximately $3.4144\times10^{14}\,\mathrm{FLOP\,s^{-1}}$; the chapter deliberately uses the rounded $3.4\times10^{14}$ rate.

Practical decisions

• Reproduce a failure on the base checkpoint before blaming SFT. If that checkpoint cannot run, probe the smallest sibling in its family.
• Put any fact with a change cadence shorter than retraining in the datastore. For fewer than roughly 100 bounded facts, a templated system prompt can beat an index on latency and operational surface.
• Use LoRA for shape and full-parameter training for capability. Frozen embeddings may not add a language or domain vocabulary the tokenizer never saw.
• Ask refresh cadence before model size. A static million-document domain on an annual cadence can justify continual pre-training.
• Keep a retrieval-free capability probe. About a dozen supplied-text arithmetic and two-hop questions can separate base-model regressions from index regressions. Run it on each base-model swap rather than each index rebuild.

4.2 Memorization: three laws and their consequences

Definition and laws

A probe samples $3{,}000$ training documents, supplies $50$ tokens, greedily decodes $50$ more and finds two exact matches. The observed sample hit rate is $2/3000\approx0.0667\%$. It describes that sampling scheme, model, prefix length and decoding rule, with sampling uncertainty; it is not a certified upper bound on extractability under other prompts. Define a sequence as extractable when some specified allowed prefix and decoding procedure reproduce the target continuation. The reported family-level trends can be written as a local empirical regression:

The positive fitted coefficients describe trends within the tested model families, duplication ranges and matching-prefix lengths, not universal laws for every aligned model. The linear-in-log form requires $N,d,k\gt 0$ and can predict values outside $[0,1]$ if extrapolated, so it is not a globally valid probability model. Repetition can strengthen memorization, and a longer matching prefix can better identify a document. Shared copying mechanisms also support in-context behavior, but that observation is not an impossibility proof that every useful intervention must destroy in-context learning.

Why the short audit fails

The $50$-token result covers only a short matching-prefix regime. A long production prompt is not automatically a long prefix of one training document: five unrelated $400$-token chunks do not form a contiguous $2{,}000$-token prefix of a single source. Audit both the actual prompt structure and targeted contiguous-prefix extraction. Alignment refusal behavior is not a deletion guarantee. The cited divergence attack recovered more than $10{,}000$ unique training examples for roughly $\$200$ in its tested setting; those observations do not supply the extraction rate for this assistant.

Audit cost

Take a $7$B generator and $100{,}000$ sampled sequences. A $500$-token prefix plus $500$-token continuation gives $10^8$ processed tokens and $1.4\times10^{18}\,\mathrm{FLOP}$. The compute proxy is approximately $4{,}117.65\,\mathrm{s}$, $1.14379$ GPU-hours and $\$2.85948$. A separate contiguous-prefix experiment with $k=2{,}000$ and the same $500$-token continuation processes $2.5\times10^8$ tokens and $3.5\times10^{18}\,\mathrm{FLOP}$: approximately $10{,}294.12\,\mathrm{s}$, $2.85948$ GPU-hours and $\$7.14869$, exactly $2.5$ times the short-probe work under the proxy. This is not a claim that five unrelated chunks provide such a matching prefix. At an assumed $1\%$ hit rate, expected yield is $1{,}000$ matching generations, costing approximately $\$0.00715$ each; uniqueness requires deduplication. Applying an independently reported tenfold deduplication effect would produce a hypothetical $100$ hits, not a guaranteed post-rebuild result. A hosted attack's $\$0.02$ per unique example has a different outcome unit and serving setup, so it does not validate this local compute price.

Practical decisions

• Probe actual production prompt layouts and separately test long contiguous prefixes of known works; total prompt length is not the matching-prefix length. Use the fixed template maximum when the product caps context.
• Deduplicate before shrinking a model. If pre-training is outside your control, keep licensed text out of later gradient updates.
• If a license permits indexing but forbids training, do not fine-tune on retrieved licensed passages. Popular passages repeat across pairs, so epoch count becomes a legal parameter. Train grounding on permissively licensed, consented or appropriately synthetic passages and keep the licensed corpus at test time.
• Report a lower bound with $k$, duplication threshold, and sample frame. Bias sampling toward highly duplicated sequences.
• Flag generated spans of roughly 50 or more tokens that match a licensed-content index. Log rather than block when verbatim extraction is the requested product.
• Measure the datastore’s duplication distribution with near-duplicate checks. A curated single-source manual is a reasonable exception.

4.3 Long-tail knowledge and co-occurrence bias

Support and wrong-answer selection

A 70-billion-parameter model correctly names Microsoft’s founders, their birth years, and the company’s founding date while missing an internal product rule documented across 40 pages. For stock keeping unit (SKU) 4471, it says 30 days instead of the true 14 days with equal confidence. This gap can produce 90 percent accuracy on head-heavy public benchmarks and 40 percent on tail-heavy product traffic. Represent a fact as subject $s$, relation $r$, and object $o$. Let $d(f)$ count pre-training documents containing both $s$ and $o$. Question-answering accuracy follows:

The fitted log form applies only for $d\gt 0$ in the observed regime; $\log 0$ is undefined. Absence of direct support does not mathematically rule out correct compositional inference or later learning. A local two-variable fit $A(N,d)\approx\alpha\log_{10}N+\beta\log_{10}d+c$ gives the following constant-accuracy relation, assuming positive fitted $\alpha,\beta$:

If support falls by a positive number of decades $u=-\Delta\log_{10}d$, the fitted compensation is $\Delta\log_{10}N=(\beta/\alpha)u$. The source's long-tail extrapolation compares models near $2\times10^{11}$ parameters with a projected $10^{18}$ and support from $10^{0.5}$ to $10^4$ documents. That spans far beyond measured model sizes; the implied slope ratio is a schematic extrapolation, not a validated engineering exchange rate:

At fixed training-token count and precision, a tenfold parameter increase multiplies the $6ND$ training proxy and parameter memory by ten; serving latency still depends on hardware and batching. A larger datastore can expand factual coverage, but incurs index, search and retrieved-token costs—it is not literally free beyond one lookup. Fivefold support is $\log_{10}5\approx0.699$ decades, and repeated crawls need not provide independent new evidence. A co-occurrence sketch $P_\theta(o\mid\mathop{\mathrm{context}}\nolimits(s))\propto\mathop{\mathrm{count}}\nolimits(s,o)$ illustrates a possible bias, not the actual decoder's full conditional distribution: relation words and wider context matter. A frequent companion can outvote a rare true relation object. The cited study reports roughly $30\%$ of its measured failures selecting a high-co-occurrence word; that percentage is not a universal error decomposition.

Retrieval crossover

A retrieved passage can override a memorized association, but no universal ratio says that one prompt occurrence defeats $10^5$ training occurrences. Wrong context can also disturb a correct answer. The cited PopQA result reports retrieval harm in a particular evaluation; a pointwise crossover depends on conditional rescue and harm rates as well as closed-book competence. Traffic weights set the value of a routing policy and capacity trade-off, not the pointwise crossover by themselves. Repeated domain training can affect freshness and retained behavior, but catastrophic forgetting is a risk to measure, not an inevitable result of every update. Route on fact/relation support, not entity popularity alone.

Routing example

The scenario handles $10{,}000$ questions per day across $5{,}000$ products. Assume a Zipf traffic law with exponent one. Using exact harmonic sums $H_n=\sum_{j=1}^n1/j$ gives $H_{50}\approx4.499205$ and $H_{5000}\approx9.094509$, hence head share $w=H_{50}/H_{5000}\approx0.494717$. The top $1\%$ of products therefore receive approximately $49.472\%$ of traffic under this assumption. Rank $50$ is an assumed routing split, not a threshold proved by $500$ labels. Take closed-book accuracy $0.82$ on the head and $0.24$ on the tail, and RAG accuracy $0.68$ on the tail. For the head, additionally assume a conditional correct-to-wrong rate $0.10$ and no wrong-to-right rescues, giving $0.82(1-0.10)=0.738$. Then always-retrieve accuracy is approximately $70.869\%$, never-retrieve accuracy $52.694\%$, and perfect head/tail routing accuracy $74.926\%$. Selective routing gains approximately $4.057$ percentage points over always retrieving; a real router needs its own confusion/error audit. Five $400$-token chunks add $2{,}000$ input tokens per retrieved query. Always retrieving adds $20$ million daily input tokens at $\$6.00$ under $\$0.30$ per million. Routing adds approximately $10.10567$ million tokens, costing approximately $\$3.0317$ per day, for approximately $\$1083.43$ over $365$ days in this input-only component. An assumed $120\,\mathrm{ms}$ retrieval-stack saving applies only to bypassed head queries; it does not establish the whole service latency distribution. This example shows why a system may do better by retrieving only for less familiar products instead of retrieving for every question. The model is assumed to answer common-product questions more accurately from memory, while retrieved information helps more with less familiar products.

• Traffic is uneven. Under the assumed Zipf law, the first $50$ of $5{,}000$ products receive approximately $49.47\%$ of queries.
• Retrieval can hurt the head. The scenario assumes head RAG accuracy $73.8\%$, including no rescues of initially wrong head answers.
• The routing split is illustrative. A production threshold needs labeled, held-out outcomes and monitoring, not popularity alone.
  • Always retrieve: approximately $70.87\%$ scenario accuracy; retrieval helps on the tail but can hurt on the head.
  • Never retrieve: approximately $52.69\%$ scenario accuracy, with weaker closed-book performance on the tail.
  • Route selectively: answer common-product questions from memory and retrieve for the tail, giving approximately $74.93\%$ scenario accuracy under the stated oracle split. Selective routing adds approximately $10.106$ million retrieved input tokens per day, versus $20$ million when always retrieving. The input-only daily cost falls from $\$6.00$ to approximately $\$3.03$. The stronger argument here is the assumed accuracy gain; cost and latency savings need the same workload and serving scope.

Practical decisions

• Stratify accuracy by $\log d$ and weight by query logs. Entity share does not size traffic cost.
• If every entity is internal, treat every query as tail and skip threshold fitting.
• Start with always-retrieve until a labeled set measures $\tau$. Routing becomes attractive when head traffic exceeds roughly one third. Refit $\tau$ when campaigns change traffic, weekly in the source scenario.
• Use popularity and relation support before model confidence. Co-occurrence bias can make the modal wrong answer highly confident. For novel or internal entities with no frequency signal, use confidence and accept its calibration risk.
• Build one co-occurrence trap per relation. Use search hit counts when direct corpus counts are unavailable.
• Put tail facts in the datastore. A small, closed, static domain on an annual update cadence is the defensible exception for repeated domain training.

4.4 Hallucination taxonomy: factuality versus faithfulness, intrinsic versus extrinsic

Two oracles and three source relations

One reviewer compares a claim with supplied evidence and measures faithfulness; another checks an authoritative current source and measures factuality. These are different oracles. Classify each atomic claim as entailed, contradicted or unaddressed by the supplied context. Let $f=P(E)$ be the entailed fraction, $q=P(T\mid E)$ the probability that an entailed generated claim is true, and $p_U=P(T\mid\neg E)$ the probability that an unentailed generated claim is true. Staleness alone does not determine $q$: old claims can remain true and fresh sources can be wrong. Closed-book task accuracy is not automatically $p_U$. With these claim-level conditional definitions, the law of total probability gives:

Grounding changes factuality at rate:

If $q$ and $p_U$ remain fixed, increasing the entailed share improves factuality when $q\gt p_U$. Real grounding interventions can change claim composition and both conditional rates, so this is not an unconditional causal guarantee. The metric gap is:

The first term is true but unsupported content; the second is false but entailed content. As $f\to1$ with fixed $q,p_U$, the signed gap tends to $-(1-q)$, not $+(1-q)$. Perfect faithfulness can still reproduce a source error. Intrinsic hallucinations contradict supplied evidence; extrinsic ones are unaddressed by it. Their causes need diagnosis rather than automatic assignment to retrieval or generation. Instruction and arithmetic errors also need separate checks.

Worked example

Audit $200$ questions. The first setup emits $1{,}000$ claims: $700$ entailed, $80$ contradicted and $220$ unaddressed. Suppose direct truth checking establishes $q=0.96$ for the entailed cohort and $p_U=0.30$ for the unentailed cohort. Then $672$ entailed and $90$ unentailed claims are true, totaling $762$ or $76.2\%$. Faithfulness is $70\%$, understating truth by $6.2$ percentage points. These conditional rates are explicit scenario measurements, not quantities inferred from corpus staleness or closed-book accuracy. The grounded-or-abstain setup declines $50$ questions and emits $750$ claims: $700$ entailed, $10$ contradicted and $40$ unaddressed. Assuming the same conditional truth rates, factuality is $(700/750)(0.96)+(50/750)(0.30)=0.916$, giving $687$ true claims. Faithfulness is $700/750\approx93.333\%$, exceeding factuality by approximately $1.733$ percentage points. Across all $200$ questions, true claims fall from $3.81$ to $3.435$ per question and false claims from $1.19$ to $0.315$. Answer coverage falls from $100\%$ to $75\%$; this differs from the source's separate $20$-point compliance illustration. This removes $175$ false claims while losing $75$ true claims. If costs per false and missing-true claim are $C_{\mathrm{wrong}}$ and $C_{\mathrm{missing}}$, the change reduces linear expected loss when $175C_{\mathrm{wrong}}\gt 75C_{\mathrm{missing}}$, or $C_{\mathrm{wrong}}/C_{\mathrm{missing}}\gt 3/7\approx0.428571$. The reciprocal $175/75=7/3$ is false claims removed per true claim lost, not the required wrong-to-missing cost ratio. At $200$ judge calls and an assumed $440{,}000$ input tokens, the input-only check costs $\$0.132$ at $\$0.30$ per million. The outside summarization study's hallucination and truth rates describe its own cohorts; they do not establish this assistant's $p_U$ or closed-book accuracy.

Practical decisions

• Report faithfulness, sampled factuality, and the intrinsic-extrinsic split. If no world oracle exists, make faithfulness the contract and corpus freshness the service obligation.
• Gate on faithfulness online. Audit factuality offline unless a high-stakes error justifies a second retrieval loop.
• Route contradictions to generation and unsupported claims to retrieval or abstention. A post-update contradiction spike may instead signal disagreeing documents.
• Sample supported claims to estimate $q$. Start with 100 entailed claims per corpus per month and increase frequency when freshness exceeds 30 days or documents have legal owners.
• Pair faithfulness with true claims per question. In regulated settings, state the precision-coverage trade and obtain explicit ownership. Require attribution for answer-bearing claims while allowing a stated policy for extrinsic background claims.

4.5 The five causes of factual error

Cause model

The paired evaluation has closed-book error $\varepsilon_0=0.38$ and RAG error $0.235$. Diagnose five operational causes: outdated data, ambiguity, retention failure, over-generalization and integration. For outcome accounting, let $S=P(\text{RAG wrong}\mid\text{closed-book wrong})$ and $\varphi=P(\text{RAG wrong}\mid\text{closed-book correct})$. These paired conditional probabilities lie in $[0,1]$. The valid error identity is:

Cause-specific diagnostic counts can change labels after retrieval. If $n_i$ initially wrong claims are assigned cause $i$ and $b_i$ remaining-wrong claims are reclassified after retrieval, the descriptive ratio $m_i=b_i/n_i$ may exceed one because claims can move between labels. Then $S=\sum_i(n_i/\sum_j n_j)m_i$, but $m_i$ is not the survival probability of the original cause-$i$ cohort. A joint cause-transition table is needed to infer that. Retrieval recall $r$ alone does not imply $S=1-r$ or $m_1=m_3=1-r$: retrieving evidence and successfully using it differ. With $S,\varphi$ held fixed and $1-S+\varphi\gt 0$, retrieval helps above the break-even error:

The threshold follows from $\varepsilon_0(1-S)\gt (1-\varepsilon_0)\varphi$. It is a comparative-static calculation with fixed paired rates; a new retriever or prompt can change both. Possible pairwise source comparisons are $\binom{k+1}{2}=k(k+1)/2$, or $15$ at $k=5$ and $210$ at $k=20$, when weights are treated as one additional source. These are counts of potential comparisons, not conflict probabilities. Operational cause labels can overlap, and an ordered diagnostic cascade makes the reported labels mutually exclusive by policy rather than proving unique causal ownership.

Audit and alternatives

Of $1{,}000$ questions, $380$ are initially wrong. Their diagnostic counts are $140,60,120,40,20$ across the five causes. Suppose paired testing finds $173$ still wrong after RAG, reclassified as $21,78,18,16,40$. The ambiguity and integration counts can exceed their original counts only because these are post-retrieval labels, not survivors confined to the same original buckets. The ratios $0.15,1.30,0.15,0.40,2.00$ are descriptive count ratios; they cannot be used as independent survival probabilities. The aggregate $S=173/380\approx0.455263$ is valid. A separate $\varphi=0.10$ produces $62$ newly wrong answers from the $620$ initially correct, so total error is $(173+62)/1000=23.5\%$ and net improvement is $14.5$ percentage points. For a counterfactual budgeting illustration only, reducing the first and third post-RAG counts from $21,18$ to $7,6$ while keeping everything else fixed gives $147+62=209$ wrong answers, a $2.6$-point improvement. Raising recall from $0.85$ to $0.95$ does not by itself guarantee these count changes. A hypothetical clarification removing $26$ of the $78$ ambiguity-labeled errors gives the same arithmetic improvement; the new descriptive ratio is $52/60\approx0.866667$, not an exact $0.87$ product. A user clarification and an automated rewrite are different interventions with different costs. Increasing generator presentation from five to twenty $400$-token chunks adds $6{,}000$ input tokens per query, $6$ million over $1{,}000$ questions, costing $\$1.80$ at the stated input rate. It does not automatically increase reranker candidate depth; if candidate depth itself changes from five to twenty, the increment is $15$ scored pairs, not $20$. Potential source pairs rise from $15$ to $210$. The paired-rate threshold is $\varepsilon_0^\star=0.10/(1-173/380+0.10)\approx0.155102$, corresponding to approximately $84.490\%$ closed-book accuracy under fixed rates.

Practical decisions

• Triage 100 failures through the four tests before changing retrieval. About two engineer-days can estimate the cause vector.
• Track ambiguity and integration separately. A single authoritative source with $k\le2$ is the main low-conflict exception.
• Set generator top-$k$ against conflict, not recall alone. Retrieve 50 candidates, rerank them, and feed 5 unless near-duplicate canonical passages justify more.
• Run the same evaluation closed-book to measure flips in both directions. A 14.5-point net gain can hide 62 newly wrong answers.
• Gate retrieval once a slice falls below roughly 15 percent closed-book error. Retrieve anyway when contractual attribution, rather than accuracy, is the requirement.

4.6 Why verification is hard: black box, moving truth, biased verifiers

Three structural difficulties

The radiology example illustrates a judge sharing the generator's mistaken impression; an expert with the right evidence can disagree. A faithfulness percentage is meaningful only with its reference, scoring unit and judge error understood. For a fixed prompt, decoder and claim-emission event, repeated trials can be modeled as Bernoulli draws only if the event is binary and the sampling assumptions hold. Positive temperature alone does not guarantee independent or varying outcomes. For nominal $95\%$ planning precision within an absolute $\pm0.05$ on that event probability, use:

For independent Bernoulli repetitions of a specified event, the normal-approximation planning formula at $p=0.5$, $z=1.96$ and absolute half-width $\Delta=0.05$ gives $n_{\mathrm{plan}}=384.16$, rounded upward to $385$. This is not an exact finite-sample coverage guarantee, and correlated samples can require a different design. Model resampling measures variation under the chosen prompt and decoder; it does not directly establish truth or locate the faulty stage. Separately, suppose changes follow a Poisson process with mean lifetime $\tau$. The probability of no change by age $t$ is $e^{-t/\tau}$, giving a maximum-age rule for tolerated probability $\delta$ of at least one change:

With $\tau=2$ years and $\delta=1\%$, $t^\star=-2\ln(0.99)\approx0.0201007$ years, or approximately $7.34$ days at $365.25$ days per year. This bounds the probability that a fact has changed since its last check under the model, not the probability that every old answer is wrong. Production truth can change at different rates or change back, so validity dates and authoritative sources still matter.

In this verifier scenario, $s$ is the fraction of initially false claims in a diagnosed blind-spot cohort that the verifier always misses; $\mu_0$ is its conditional miss probability on the remainder. The formula assumes detected false claims are blocked and verification introduces no new false claims. Its denominator is all original claims, not only accepted claims. A better judge may change both $s$ and $\mu_0$ in practice; holding $s$ fixed illustrates a structural floor, not a universal limit on model upgrades. False-positive rejection and emitted-claim coverage need separate measurement. Unsettleable claims should be excluded from both scored numerator and denominator, with their share reported, rather than silently marked true or false.

Verifier audit and costs

Audit $1{,}000$ claims, $200$ of which are false. Suppose an independent diagnostic—not the judge's pass count alone—identifies $120$ structural blind spots, giving $s=0.60$. Assume miss probability $\mu_0=0.15$ on the other $80$ false claims. Expected false passes are $120+80(0.15)=132$, or $13.2\%$ of all original claims. A single observed pass fraction cannot separately identify $s$ and $\mu_0$: if only $120$ of $200$ passed and $\mu_0=0.15$, the implied scenario value would instead be $(0.60-0.15)/(1-0.15)\approx0.5294$. Now suppose an independent dated-source check works specifically on the blind-spot cohort with on-target probability $0.70$ and conditional correctness $0.95$, and that this resolves those cases. The usable-check probability is $0.70(0.95)=0.665$, so the residual blind-spot share is $s'=0.60(1-0.665)=0.201$. Keeping $\mu_0=0.15$ on all newly catchable cases gives expected residual false claims $200[0.201+0.799(0.15)]=64.17$, or $6.417\%$ of the original claims. Fractional counts are expectations, not rounded observed claims. The expected gain over $132$ is $6.783$ percentage points; eliminating only independent misses at fixed $s=0.60$ offers at most $1.2$ points, a ratio of approximately $5.65$ under these assumptions. A small judge with $800$ input tokens per claim costs $\$0.24$ for this input-only component at $\$0.30$ per million. One extra $400$-token passage per claim adds $\$0.12$, totaling $\$0.36$ before search, embedding and other serving costs. A $\$3$-per-million judge at the original input length costs $\$2.40$, approximately $6.67$ times that scoped token bill; its actual effect is not established by the price. Macro-$F_1$ of $0.8$ does not imply a $0.2$ conditional false-claim miss rate. Only if a separate miss probability $0.20$ is explicitly assumed would an independent model predict $200(0.20)=40$ false passes; $132/40=3.3$ compares two assumed models, not a deduction from $F_1$. If $300$ labels have $\tau=2$ years, expected labels with at least one change after six months are $300(1-e^{-0.25})\approx66.36$. This is $6.636\%$ potentially affected labels, not a measured $6.636$-point score bias: the net accuracy effect depends on which labels changed, whether old answers remain valid and model behavior.

Practical decisions

• Estimate $s$ before upgrading a judge. Replay confirmed false claims to measure total misses, then use an independent diagnostic or additional interventions to separate structural blind spots from conditional misses.
• Give the verifier a retrieved passage with a web address and timestamp that the generator did not have. Supplied context is the reference for faithfulness, but a judge can still misread it; shared access is not statistical independence.
• Check information-access symmetry. Roughly one quarter of attribution-judge errors arise when judges and annotators see different amounts of source text.
• Stamp gold labels with validity dates and lifetime classes. Recheck the $\tau = 2$ year, 1 percent-tolerance slice weekly, stable facts annually, and fast-changing prices, headcounts, or org charts daily.
• Report claim emission rates at production temperature. Start with five samples per item because three emissions in five differ from five in five. Use temperature zero only for explicitly labeled regression gates.
• Send clinical, legal, and financial claims to experts when generator and judge share a distribution and $s\to1$. Use an authoritative curated source instead when one exists.

Diagrams

Figure 4.1

Diagram description and source

flowchart TB
    A["Pre-training: broad capability and uneven factual memory"]
    B["SFT and PEFT: task behavior, format and learned context use"]
    C["Post-training: preference, reward-guided behavior and possible new learning"]
    D["Datastore: addressable sources, freshness and keyed deletion"]
    A --> B
    B --> C
    C -. "retrieval adds a separate update path" .-> D

Figure $4.1$: Stage labels are diagnostic starting points, not hard mathematical ceilings. Training and retrieval can each change behavior, but a keyed datastore preserves a different update and deletion mechanism from shared parameters.

Figure 4.2

fraction k-extractable, log scale
^
|          x10 duplication                              6B
|              ^                           ____________/
|              |                 __________           1.3B
|                    - - - - - -
|          . . . . . . . . . . . . . . . . . .       125M
|
+----------|------------|----------|----------------|----> context tokens k, log scale
          50           200        500             2,000
       audit probe                     matching source prefix
          [ range a probe at k = 50 never measures ]

Figure $4.2$: The cited experiments report increasing extractability with model scale, duplication and matching-prefix length within their measured regimes. The axes and curves are schematic. A short probe does not characterize longer source-prefix attacks, and a long multi-document prompt need not be one matching training prefix.

Figure 4.3

answer accuracy (%)
^
| 80                                      / weights only
|                                      __/ | answers retrieval flips wrong
|                         . . . . . . X . . weights plus retrieval
| 60              . . . .             | tau
|           . . .                     |
| 40                         _________/
|                    _______/
| 20          ______/
|
+---------|---------|---------|---------|---------|----> supporting documents d, log scale
         10^0      10^1      10^2      10^3      10^4
         retrieve                         trust the weights

Figure $4.3$: This is a schematic crossover, not a universal accuracy curve. Low direct support can reduce parametric recall without proving zero ability to infer an answer. Retrieval can help or harm depending on evidence and the generator. A policy cutoff should be fitted to paired outcomes and the workload's utility, not inferred from popularity alone.

Figure 4.4

Figure $4.4$: Faithfulness grades the columns and factuality grades the rows, so the two verdicts differ in three of the six cells - most dangerously in the faithful-but-false cell directly below the shaded one, which no faithfulness metric can ever see. Each cell names a different owner, which is the taxonomy’s operational value.

Figure 4.5

Diagram description and source

flowchart TB
    A["One wrong atomic claim"] --> B["1. Re-ask with ambiguity resolved. Fixed?"]
    B -- yes --> C2["Cause 2: ambiguous prompt"]
    B -- no --> D["2. Was the wrong answer true earlier, or asserted by the corpus?"]
    D -- yes --> C1["Cause 1: outdated training data"]
    D -- no --> E["3. Paste the answer-bearing passage. Fixed?"]
    E -- yes --> C3["Cause 3: retention or knowledge-use limitation"]
    E -- no --> F["4. Does a second prompt passage contradict the first?"]
    F -- yes --> C5["Cause 5: knowledge integration"]
    F -- no --> C4["Cause 4: over-generalization"]
    classDef initial stroke-width:2px
    classDef shrinks stroke-width:2px,stroke-dasharray:5 5
    classDef grows stroke-width:3px
    class A initial
    class C1,C3,C4 shrinks
    class C2,C5 grows

Figure $4.5$: Ordered tests assign one operational label per audited failure; they do not prove unique causal ownership. Compare the same diagnostic order across systems and retain paired outcome cohorts. Cause growth can reflect relabeling, so it is not a greater-than-one probability that an original error survives.

Figure 4.6

Diagram description and source

flowchart LR
    A["200 false claims"] --> B["Independently diagnosed blind spots: 120; catchable remainder: 80"]
    B --> C["Assumed miss rate on catchable claims: 0.15"]
    C --> D["Expected false passes: 132; fixed-blind-spot floor: 120"]
    A --> E["Dated source resolves 66.5% of the diagnosed blind-spot cohort"]
    E --> F["Expected blind spots: 40.2; catchable: 159.8"]
    F --> G["At the same conditional miss rate: 64.17 expected false passes"]

Figure $4.6$: Under the stated fixed-cohort model, the original verifier leaves $132$ false claims and dated sources reduce the expectation to $64.17$. The $6.783$-point expected improvement exceeds the $1.2$-point headroom from eliminating only independent misses at fixed blind-spot share. These outcomes require the conditional source-success assumptions; neither shared model family nor a single pass rate identifies that share.

Whiteboard pack

Numbered drawing order

1. Draw a vertical stack labeled pre-training, SFT, and post-training.
2. Write capability beside pre-training, shape beside SFT, and preference beside post-training.
3. Draw a dotted boundary under gradient descent.
4. Add a datastore below it with freshness, provenance, and deletion.
5. Draw two accuracy curves against fact support $d$. Mark their crossover $\tau$.
6. Label the left side retrieve and the right side trust the weights.
7. Draw a two-row by three-column grid for factuality and faithfulness.
8. Add the five-cause diagnostic cascade beside the grid.
9. Finish with two verifier boxes. Give one the same model family and the other a dated source.
10. Circle the shared blind-spot floor and write oracle independence first.

90 to 100 word script

Start by separating what lives in the model from what lives outside it. Pre-training gives the model capability and uneven memory. Supervised fine-tuning shapes how it answers, and post-training shapes preferences and abstention. A datastore owns facts that change. Retrieval helps rare facts, but it can overwrite a correct memory with a bad passage. So I grade claims on two axes: faithful to the supplied context and factual in the world. Then I bucket errors by cause. Finally, I verify with dated evidence the generator did not see, because a stronger judge cannot detect a blind spot both models share.

Interview traps

1. Would you fine-tune on a new price list?

No, unless the prices are static for longer than the retraining cycle. SFT fixes output shape and grounding behavior, while a datastore fixes freshness cheaply and supports immediate deletion.

2. A 50-token probe found 0.07 percent extractability. Is licensed content safe?

The observed $2/3000\approx0.0667\%$ is a finite-probe sample rate, not a safety guarantee. Test actual prompt layouts and longer contiguous prefixes of known sources separately; five unrelated chunks are not a single $2{,}000$-token document prefix. Deduplication and deployment-length probes can reveal risk, but a failed extraction search does not prove absence or settle permission to use the work.

3. When should retrieval be skipped?

Skip it on traffic where closed-book memory already beats retrieval, or where closed-book error falls below the measured break-even. An off-target passage can flip a correct head answer, so measure the popularity crossover and route by fact support rather than retrieve everywhere.

4. Does higher faithfulness mean higher truth?

Under fixed claim composition, factuality rises with the entailed fraction when $q=P(T\mid E)$ exceeds $p_U=P(T\mid\neg E)$. Those are not automatically corpus freshness and closed-book accuracy. Perfect faithfulness can still repeat false source claims, so report source-based support, independently checked truth and answer coverage separately.

5. Should verification budget buy a stronger judge or dated sources?

Measure the verifier's actual blind spots and conditional misses before upgrading it. In a model holding the blind-spot cohort fixed, improving only the remaining miss rate has limited headroom; independent authoritative sources can resolve some shared errors. Real model upgrades may change both terms, so compare measured outcomes rather than assuming source access always beats judge capacity.

Key numbers

00-START-HERE · 01-READING-ROUTES · 03_a_repeatable_framework_for_any_rag_design_question · 05_data_privacy_and_the_legal_surface