Chapter 1: What a RAG Interview Actually Tests

00-START-HERE · 01-READING-ROUTES · 00_09_about_the_author · 02_the_competing_answers_to_the_same_problem Retrieval-augmented generation (RAG) interviews test whether you can route knowledge to the right store, price that choice, and recognize when retrieval is the wrong tool.

TL;DR

• A language model is a fixed-capacity knowledge store. A datastore grows as rows are added. RAG joins those two stores.
• The decisive difference is write cost. A datastore row is addressable and reversible. A fact spread through model weights is entangled and expensive to change.
• Five bottlenecks follow from storing knowledge in weights: factual error, no source verification, high update cost, no bounded opt-out, and model size tied to factual coverage.
• Five RAG benefits move those responsibilities into a datastore: correctness, attribution, opt-out, adaptability, and parameter efficiency. Each move creates a retrieval-side cost.
• RAG is one special case of tool use. Its call is read-only, retryable, and returns passages with source identifiers.
• Retrieval cannot answer queries that require every matching record when it returns only the top few records. Aggregates and actions need different tools.
• Retrieval can harm answers the model already knows. Gate it by measured subject popularity or a product-specific frequency proxy once the crossover is known. In the worked popularity example, selective retrieval reaches $80.1989\%$ accuracy, beats always-retrieve by $4.1489$ percentage points, and cuts retrieval work by $55\%$.

The story

Imagine an interview room as a working library with two places to keep knowledge. The first place is a stone memory wall. Facts are carved across many stones at once. This is the parametric store, meaning its capacity was fixed before training. Reading it is fast, but changing one fact means recarving a wide surface. The second place is an expandable card room. Each card has an address and can be inserted, corrected, or removed. This is the non-parametric datastore, meaning its knowledge capacity grows as cards arrive. RAG is the librarian who uses both rooms. The librarian reads the question, retrieves a few cards, and asks the memory wall to compose an answer from the question and those cards. That combination is semi-parametric, meaning one fixed store works with one expandable store. The interviewer first inspects the memory wall for five cracks. Rare facts fade because the carving is lossy. A sentence has no return address to its source. A correction requires recarving. A deletion cannot target one original document. More factual coverage also means paying for a larger wall on every visit. Moving facts into the card room repairs those cracks, but the bill moves too. A missing card can spoil correctness. A citation can point to the wrong support. A deletable card helps only if the same text was never carved into the wall. Fresh cards require an ingestion schedule. A smaller wall now depends on room space and an extra reading step. The library also has other service desks. The retrieval desk returns source cards and is safe to ask twice. The calculator desk returns a value but no source passage. The action desk opens a ticket or sends a message and must not be called twice by accident. RAG is one desk, not the whole library. Some questions ask for a few relevant cards. Other questions ask for the sum of every invoice. A top-card desk cannot guarantee a complete total, even with perfect ranking. The interviewer wants you to route the first query to retrieval and the second to a complete query tool. Finally, a doorman decides whether the librarian should enter the card room at all. Obscure subjects benefit because the wall remembers them poorly. Famous subjects can be harmed by a distracting card because the wall already knows the answer. The strongest interview answer therefore sounds like a library operating plan. Name the two stores. Price each write. Trace the five cracks and five transfers. Route each query to the right desk. Then place a measured gate at the point where retrieval stops helping.

Decoder table

Core mechanics

Mathematical conventions

Storage is decimal unless marked binary: $1\,\mathrm{GB}=10^9\,\mathrm B$ and $1\,\mathrm{MB}=10^6\,\mathrm B$. FLOP estimates use $W_{\mathrm{forward}}\approx2NT$ and $W_{\mathrm{train}}\approx6NT$ for parameter count $N$ and processed tokens $T$. At assumed sustained throughput $P_{\mathrm{GPU}}=3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$, $t_{\mathrm{ideal}}=W/P_{\mathrm{GPU}}$ on one GPU. These are arithmetic proxies, not measured latency: they omit attention-length effects, bandwidth, communication, batching, index I/O and rollout. Multi-GPU examples assume ideal scaling. Training epochs, replay budgets and routing rates are scenario inputs, not universal requirements. Calculate at full precision, then round; use $\approx$ for rounded or modeled results.

1.1 Parametric, non-parametric, semi-parametric: the frame behind every question

What the frame is

A parametric language model fixes its parameter count $\lvert\theta\rvert$ before training. That count does not grow with the amount of training data. The model represents the next-token distribution below over a vocabulary of size $V$.

After training, the source corpus is no longer an addressable part of the model. Everything retained from that corpus is compressed into the weights. A non-parametric datastore grows with its contents. Store $N$ chunks and its knowledge capacity is $\mathcal O(N)$. Add chunk $N + 1$ and capacity grows without retraining or disturbing an existing row. Dense retrieval is k-nearest-neighbor search in embedding space. It is non-parametric in capacity even though its encoder can contain tens or hundreds of millions of trained weights. RAG composes the two stores.

The query is $x$. The datastore is $D$. The retrieved chunks are $c_1$ through $c_k$. The answer is $y$.

Why the split exists

The split earns its keep through write-cost asymmetry, not through an unconditional accuracy claim. Writing one fact to $D$ embeds one chunk and inserts one row. That write takes milliseconds in the prose example, leaves other rows unchanged, and is reversible by deletion. Writing one fact to $\theta$ requires a gradient pass over the parameters. For a 27-billion-parameter model held in FP32, the weights alone occupy $(27\times10^9)(4\,\mathrm{B/parameter})=108\,\mathrm{GB}$ before optimizer state. The update is smeared across the weight tensor and cannot be reversed by subtraction. Knowledge in the weights is effectively write-once and entangled. Knowledge in the datastore is addressable and revocable. Freshness, attribution, opt-out, and tenant isolation follow from that difference. The durable default is behavior in the weights and knowledge in the datastore. Format, instruction following, refusal, and reasoning are learned dispositions. Facts, prices, policies, and tenant data are rows that may need a Tuesday correction.

What breaks without the frame

Putting changing facts into the weights makes corrections depend on another training run. Putting behavior into retrieval does not guarantee that the model will emit a required format such as JSON. The maximalist datastore design also fails. A tiny generator must still resolve references across $k$ passages, reconcile contradictions, and notice missing information. A larger datastore increases that reading burden through near-duplicates and conflicting passages. Knowledge load and comprehension load are different. The mirror design that folds retrieval back into the weights also fails the write-cost test. It restores retraining as the update primitive.

Compression arithmetic

The source uses Llama 3 to quantify the lossiness. It reports more than 15 trillion pre-training tokens and an 8-billion-parameter model. That is about $1{,}875 \,\mathrm{\,\mathrm{tokens}/parameter}$. English text is approximated at 4 bytes per token. A bf16 parameter occupies 2 bytes. At the rounded $15\times10^{12}$-token reference scale, the input-to-parameter ratio is $15\times10^{12}/(8\times10^9)=1{,}875$ tokens per parameter. The corresponding input-to-checkpoint byte ratio is $(1{,}875\times4)/2=3{,}750{:}1$. This is a corpus-size comparison, not a claim that each parameter independently stores $7{,}500$ identifiable source bytes or that every fact is compressed losslessly. Rare facts can be poorly retained, but their learning probability cannot be inferred from this ratio or from a count of three appearances alone.

Worked example: 40,000 internal documents

The corpus has $40{,}000$ documents at $2{,}000$ tokens each. That is $80{,}000{,}000$ tokens. About $6\%$ changes each quarter. The generator has 8 billion parameters. The parametric route uses the training-cost identity below.

The factor $6$ approximates $2$ forward-pass and $4$ backward-pass FLOPs per parameter per token. One epoch costs $3.84\times10^{18}\,\mathrm{FLOP}$. The worked scenario budgets $10$ epochs, totaling $3.84\times10^{19}\,\mathrm{FLOP}$; neither successful memorization nor the necessity of ten epochs follows from that budget. A rounded dense bf16 peak near $10^{15}\,\mathrm{FLOP\,s^{-1}}$ at $34\%$ utilization gives the assumed $3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$. The sustained throughput carried by the source is $3.4 \times 10^{14} \,\mathrm{FLOP\,s^{-1}}$. Training therefore takes $1.13 \times 10^{5} \,\mathrm{s}$, or about $31.4 \,\mathrm{GPU\!-\!hours}$. That cost repeats each quarter. The whole corpus is replayed because training only on the changed $6\%$ invites catastrophic forgetting. The datastore route chunks at 300 tokens. Use non-overlapping $300$-token chunks without crossing document boundaries, retaining a final partial chunk. Each modeled document produces $\lceil2{,}000/300\rceil=7$ chunks, for $280{,}000$ vectors. A forward-only embedding proxy for the $110$-million-parameter encoder is $1.76\times10^{16}\,\mathrm{FLOP}$, giving approximately $51.76\,\mathrm{s}$ at the stated throughput. The previously quoted $44\,\mathrm{s}$ is not reproduced by those inputs. At $768$ fp32 dimensions, vector payload is $280{,}000\times768\times4=860{,}160{,}000\,\mathrm B=860.16\,\mathrm{MB}$, excluding the index, identifiers, metadata and source text. Padding and actual encoder throughput can add work. The quarterly $6\%$ refresh touches $4.8\times10^6$ source tokens, giving approximately $3.11\,\mathrm{s}$ of embedding arithmetic before index work. Comparing the initial ten-epoch training run with the initial full-corpus embedding pass gives approximately $2{,}181.8$ times the compute. The independent check is $(6/2)(8\times10^9/(1.1\times10^8))10$. Comparing a full replay training run with a changed-only $6\%$ embedding refresh instead gives approximately $36{,}364$ times. These are different denominators; do not label the initial-pass ratio as the incremental-refresh ratio.

Practical decisions from the frame

• Route facts to $D$ and behavior to $\theta$ by default.
• Keep a required disclaimer or fixed schema in behavior when it must appear even if no query asks for it.
• Choose the store by change frequency and correction deadline. If a correction must land within days, weights are the wrong store.
• Use weights for information that is genuinely static over the model’s service life.
• Size the generator by comprehension load. Measure grounding and conflict resolution at fixed $k$ before shrinking it.
• Keep licensed text, PII, and data with deletion obligations in the datastore.
• Put data into weights only when it is provably unencumbered and its distribution is what training should teach.
• Name the routing frame aloud before proposing architecture. If the interviewer has fixed the split, reason within that constraint.
• For an invented SKU, move the changing catalog to $D$, keep fine-tuning for response format, and use exact-match signals for the identifier field.
• A store growing from 2 million to 200 million chunks does not by itself justify shrinking a 70-billion-parameter generator to 8 billion. Test the increased reading load.
• If a licensed corpus must leave within 30 days, separate the claimed four-point language-competence loss from retrievable knowledge and keep the encumbered material deletable.

1.2 The five bottlenecks that created RAG

One storage choice, five consequences

The source’s interview brief has $40{,}000$ policy documents. A legal team audits every answer. A compliance officer can require one customer’s data to be deleted within 30 days. The team has a 27-billion-parameter open-weight model and eight accelerators. Saying only that fine-tuning hallucinates is incomplete. Even a correct parametric answer cannot identify its source document, guarantee deletion, or avoid retraining cost.

Bottleneck 1: factual error

Weight storage is frequency-weighted and lossy. Accuracy rises with the number of pre-training documents that mention a question’s subject. Rare enterprise facts occupy the weak tail. Adding corpus volume is not proportional to adding distinct facts because web data is heavily correlated. Scaling parameters also fails as a practical tail strategy. The cited trend extrapolates human-level long-tail accuracy to about $10^{18} \,\mathrm{parameters}$. At 2 bytes per bf16 parameter, the weights would occupy $2 \times 10^{18} \,\mathrm{B}$. Holding them would require $2.5 \times 10^{7}$ accelerators with $80\,\mathrm{GB}$ each. That is 25 million cards for inference weights alone. The $10^{18}$ estimate is an extrapolation from a fitted curve, not a measurement. An orders-of-magnitude comparison must name its baseline. Relative to the worked $8\times10^9$-parameter model, $10^{18}/(8\times10^9)=1.25\times10^8$, about $8.10$ orders of magnitude. Describing the same target as simply “six” or “nine” orders beyond deployed models is not a reproducible comparison without a specified model size and date. The extrapolation itself remains highly uncertain.

Bottleneck 2: no verification

A fact is a diffuse perturbation across weights, not a value stored at a document address. A sampled token carries no return address to the document that influenced training. Asking the model to cite does not reconstruct provenance and can produce a confabulated citation. The weight tensor is also undated. The correct answer to a leadership question can move from one person to another while the stored tendency stays stale. The source illustrates that drift with a leadership answer moving from Jack Dorsey to Linda Yaccarino and then to another successor.

Bottleneck 3: update cost

Full fine-tuning with Adam in FP32 costs $4N$ bytes for weights, $4N$ for gradients, and $8N$ for optimizer moments. The total is $16N \,\mathrm{B}$ before one activation. At $N = 27\times10^{9}$, that is $432\,\mathrm{GB}$. The forward-pass weights alone are $108\,\mathrm{GB}$. Training only on a changed document can damage retained behavior; the outcome depends on the update method and data. Replay is one mitigation for forgetting, not a mathematical requirement for every successful update; retention must be evaluated independently.

Bottleneck 4: no bounded opt-out

There is no operation that deletes one source document from a tensor. The cited memorization results establish three regularities. Larger models memorize more. Repeated examples are more extractable. Extractability rises as an attacker supplies more context tokens. Unlearning reverses the training sign to punish specified sequences. It requires enumerating every target sequence in advance. The core problem is that the sensitive content in a large corpus is abundant and not fully known.

Bottleneck 5: model size

Training compute couples knowledge capacity to language competence. In this formula, $N$ is parameter count and $D$ is training-token count rather than the datastore.

Increasing parametric capacity to improve factual coverage raises work on every forward pass, including queries that need no external facts. Adding one fact does not automatically increase a fixed architecture's parameter count. Scaling can reduce some factual error; it does not itself provide document-level provenance or deletion. Model editing also needs locality and regression checks because changing shared parameters can affect unrelated behavior. Changing neurons associated with one fact can perturb unrelated knowledge and leave contradictory beliefs.

Worked example: one policy changes on Tuesday

The setup uses $40{,}000$ documents, a $27$-billion-parameter model and eight $80\,\mathrm{GB}$ accelerators. Each sustains the assumed $3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$. FP32 Adam weights, gradients and two moments together use $16N=432\,\mathrm{GB}$ before activations. Optimizer moments alone are $8N=216\,\mathrm{GB}$. Aggregate capacity therefore needs at least $\lceil432/80\rceil=6$ cards for those tensors alone; this does not mean six cards are reserved and two are free. Perfectly sharding the state across eight cards uses $54\,\mathrm{GB}$ per card before overhead. The replay budget is $10^9$ tokens. The replay proxy is $1.62\times10^{20}\,\mathrm{FLOP}$. With ideal aggregate throughput $2.72\times10^{15}\,\mathrm{FLOP\,s^{-1}}$, elapsed time is approximately $59{,}558.8\,\mathrm{s}=16.54$ hours. The run consumes approximately $132.35$ GPU-hours and costs approximately $\$330.88$ at $\$2.50$ per GPU-hour, before regression, communication and rollout. For model editing, the source assumes about 50 atomic statements per document. The corpus then contains $2 \times 10^{6}$ facts. At $3\%$ annual churn, it receives $60{,}000$ edits per year, or 164 per day. Each edit needs a regression check because it can perturb neighboring knowledge. The arrival rate, not the cost of one edit, is the bottleneck. For retrieval, this separate update scenario uses an $8{,}000$-token document—an outlier relative to the earlier $2{,}000$-token average—split into twenty $400$-token chunks. A $110$-million-parameter bi-encoder processes them in the proxy $1.76\times10^{12}\,\mathrm{FLOP}$, or approximately $5.18\,\mathrm{ms}$, plus index upsert work. Comparing the eight-GPU replay wall time with this one-GPU embedding-only time gives approximately $1.15\times10^7$. This is an idealized, differently resourced wall-time ratio, not a measured end-to-end update speedup. The later deletion is a row removal. The compute identity is checked against GPT-3. For 175 billion parameters and 300 billion tokens, it predicts $3.15 \times 10^{23} \,\mathrm{FLOP}$. The reported training compute is $3.14 \times 10^{23} \,\mathrm{FLOP}$. The 16.5-hour estimate therefore depends mainly on the replay budget assumption.

Practical decisions from the five bottlenecks

• Run all five bottlenecks against every proposal to put company documents into weights.
• Use fine-tuning for stable, always-needed, non-attributable behavior such as house style, output format, and domain jargon.
• Price memory with $16N$ and training compute with $6ND$ before rejecting the parametric route.
• Below roughly 1 billion parameters, a full fine-tune can take hours on one card. The argument then shifts toward attribution and opt-out.
• Keep customer data, licensed corpora, and retention-controlled data out of weights by default.
• Deleting a datastore row does not remove text memorized during base-model pre-training.
• A stronger model can help high-popularity closed-book accuracy. It does not solve bottlenecks 2 through 5.
• Size update systems by arrival rate. For facts that change annually or less, a scheduled retrain may be simpler.
• A million-token context can paste in a changed policy and keep its source text visible. It also turns index work into per-query prefill and does not solve every opt-out concern.
• Cheap fine-tuning addresses factual fit but leaves attribution, deletion, and coupled serving size intact.
• The long-context argument flips only if prefilling that million-token context costs less than the ranked retrieval stage it replaces.

1.3 The five benefits, and what each one costs you

One responsibility moves five times

The generation step can be written as a conditional answer over the query and retrieved chunks.

The weights are learned and then frozen for service. The datastore is queried at inference time. Each claimed RAG benefit transfers one responsibility from the weights to that datastore. The responsibility does not disappear. It acquires a new operational failure surface.

Benefit 1: factual correctness

The fact no longer has to survive pre-training compression. Rare entities that appeared only a few times can be supplied directly from a document. The cost is placing retrieval on the critical path. A wrong passage can override a parametric answer that would have been correct. The cited PopQA comparison measures this reversal on roughly $10\%$ of questions. Retrieval miss and retrieval-induced harm are therefore part of the correctness budget.

Benefit 2: attribution

Retrieved chunks carry document identifiers, so generated claims can point back to supplied sources. The source names inline citation markers and GopherCite’s verified quotes as examples. Purely parametric generation offers no equivalent path from a token to one training document. The cost is that a citation marker is a provenance claim, not proof of support. A generator can cite a passage it did not use or a passage that does not entail the sentence. GopherCite samples multiple candidate answers and reranks them against quoted spans. Honest verified attribution therefore multiplies generation cost by the sample count.

Benefit 3: data opt-out

If a document exists only in $D$, removal is a row deletion. If it has entered $\theta$, removal becomes the sequence-enumeration problem of unlearning. The cited low-risk split trains weights only on permissively usable text. Copyrighted, private, and attribution-required material stays in a test-time datastore that can be amended or dropped. The cost appears in the cited perplexity tables. A model trained only on public-domain text is a weaker language model. Retrieval must repay that quality gap on every query, including queries that need no facts.

Benefit 4: adaptability

A current officeholder can change without a gradient step. The price is an index with a freshness objective, an ingestion pipeline, and a rebuild cadence. The stale parametric prior remains underneath the new passage and can contradict it.

Benefit 5: parameter efficiency

A generator that does not need to memorize the world can be smaller. The cited Min et al. result shows a retrieval-augmented model matching a much larger parametric RoBERTa on the two senses of cheap. The retrieved neighbors preserve the distinction between inexpensive and poor-quality construction that the weights had collapsed. The price is index memory and a second latency stage. Continual fine-tuning does not reproduce all five benefits. It has no source attribution because gradients erase provenance. It has no bounded delete because targets must be enumerated. It suffers from catastrophic forgetting on updates and may require synthetic replay from the original model.

Worked example: 27 billion parameters versus 7 billion plus an index

The documentation corpus contains $10^{7}$ chunks at 768 dimensions. Serving weights use bf16 at 2 bytes per parameter. A 27-billion-parameter model occupies $54\,\mathrm{GB}$ of accelerator memory. In FP32, it would occupy $108\,\mathrm{GB}$. On an $80\,\mathrm{GB}$ H100, the BF16 model leaves about $26\,\mathrm{GB}$ for KV cache and activations. A second independent $27$B bf16 checkpoint would not fit alongside the first on one $80\,\mathrm{GB}$ card, though tenants can share one model; tenant count is not itself a checkpoint count. A $7$B generator uses $14\,\mathrm{GB}$ of parameter payload, freeing $40\,\mathrm{GB}$ of HBM. The raw fp32 vectors use $10^7\times768\times4=30.72\,\mathrm{GB}$. The HNSW example uses $M=16$. With up to $2M=32$ four-byte neighbor identifiers per node, HNSW base-layer link allocation is $1.28\,\mathrm{GB}$. Vectors plus these base-layer links total $32.00\,\mathrm{GB}$. This is not an all-in index size: upper layers, identifiers, allocation overhead, metadata and source text are omitted. The trade is also across memory media, not simply $40\,\mathrm{GB}$ saved against $32\,\mathrm{GB}$ spent in the same pool; the saved parameter bytes are accelerator HBM. The index budget is host DRAM on a CPU machine. The source’s order-of-magnitude price comparison with HBM is a pricing assumption, not a fixed hardware law. At $96\,\mathrm{B/vector}$, IVF-PQ codes for $10^7$ vectors occupy $0.96\,\mathrm{GB}$, exactly $32$ times smaller than the $30.72\,\mathrm{GB}$ raw-vector payload. This excludes codebooks, identifiers, coarse centroids, inverted lists and source text. Retrieval also adds prompt work. At $k = 5$ chunks and 400 tokens per chunk, retrieval adds $2{,}000$ prompt tokens. A forward pass is approximated as $2PT \,\mathrm{FLOP}$ for $P$ parameters and $T$ tokens. For the 7-billion-parameter generator, prefill costs $2.8 \times 10^{13} \,\mathrm{FLOP}$. At $3.4 \times 10^{14} \,\mathrm{FLOP\,s^{-1}}$, that is 82 milliseconds. The vector arithmetic is checked against a shipped DPR index. Twenty-one million Wikipedia passages at 768 float32 dimensions require $64.5\,\mathrm{GB}$. The DPR release reports roughly $65\,\mathrm{GB}$.

Practical decisions from the benefit ledger

• Compute $N \times d \times 4$ before choosing index compression.
• Move to IVF-PQ or scalar quantization when flat vectors exceed roughly $60\%$ of node memory. The $30.72\,\mathrm{GB}$ raw-vector payload alone reaches a $60\%$ budget at $30.72/0.6=51.2\,\mathrm{GB}$ of host capacity; it is $48\%$ of a $64\,\mathrm{GB}$ host. Add index and application overhead before selecting a machine.
• Use chunk-level citations by default because they add one response field and no extra generation.
• Use verified-quote decoding when an unsupported sentence creates liability in regulated medical, legal, or financial output.
• Budget sample-and-rerank as a generation multiplier, not as a free citation feature.
• Keep the generator’s broad parametric knowledge unless licensing is the binding constraint.
• Use low-risk-only training when a contract or jurisdiction requires proof that specified text never entered training.
• Gate retrieval instead of calling it on every query. This avoids harmful flips and saves the 82-millisecond prefill in the example.
• Retrieve unconditionally when the corpus is private and the model has no parametric coverage, because the gate has no correct skip path.
• State ingestion lag, rebuild cadence, and mid-rebuild reader behavior.
• An append-only immutable corpus, such as statutes or published papers, can use a simpler nightly incremental build.

1.4 RAG is one instance of tool use, not the general case

The formal boundary

A RAG system first retrieves passages and then generates from them.

A tool-augmented generator generalizes that sequence by choosing a tool and arguments before producing an answer.

A simple single-retriever RAG pipeline is the special case $\lvert T\rvert=1$, with $\mathop{\mathrm{RET}}\nolimits$ as its sole tool. This is a modeling example, not a definition restricting all RAG systems to one tool: hybrid search, multiple retrievers and agents can still perform retrieval-augmented generation. The retrieval argument is the query or a reformulation; its evidence return contains passages and source identifiers. The relevant boundary here is that evidence contract, not the number of tools or the index technology. BM25 over web pages can satisfy the RAG contract. A search endpoint that returns only a synthesized answer string does not preserve it.

What general tool use adds

Tool selection turns a fixed retrieval pipeline into a policy. When $\lvert T\rvert \gt 1$, a classifier, router, or learned policy must choose $t_i$. The source names Toolformer as a self-supervised tool-selection approach. It names ReAct as an approach that interleaves reasoning and actions in the trace. It also notes that retrieval research can cast answer-versus-search as a reinforcement-learning policy. Argument synthesis produces a query or payload before the call runs. A weak retrieval query usually degrades into poor ranking. A malformed SQL or JSON argument can fail to parse. A syntactically valid argument with the wrong date range can return a plausible but wrong number. Effects remove cheap retries. Calling read-only retrieval twice adds latency but no second state change. Calling a ticket-creation tool twice can create a duplicate action.

Why retrieval has a distinct contract

The generator already knows how to read text in context, so passages require no new model capability. An answer span can be traced to a retrieved chunk and then to a source document. A human can audit the answer against those passages without trusting the model’s hidden computation. A calculator that returns 42 has a different contract. Its warrant is the calculator’s correctness, not a cited document. RAG returns evidence that can be cited. Other tools often return values that must be trusted. The evidence-returning, state-mutating quadrant is nearly empty. A retrieval call that logs a query or decrements a quota remains citable but is no longer free to retry. That distinction matters when a low-confidence loop re-queries.

Why indexing everything fails

A relevance index returns a few passages; an aggregate or exhaustive list can depend on every matching record. If the answer needs $M\gt 0$ distinct records and retrieval returns $k$ records, evidence coverage is bounded above by $\min(k,M)/M$, and by $k/M$ when $M\gt k$. Equality requires every returned record to be relevant and distinct. A strong relevance ranker does not thereby certify completeness. The source calls such corpus-wide demands global or sensemaking queries.

Worked example: route 1,000 production queries

The traffic mix has $620$ lookups, $180$ aggregates, $120$ actions and $80$ closed-book queries. In this scenario, assume a retrieval-only product succeeds on every lookup and closed-book query, fails on these aggregates, and has no action execution path. Its resulting upper bound is $(620+80)/1{,}000=70\%$. This is a conditional workload ceiling, not a universal ceiling for RAG; a product with complete aggregation or action tools changes the assumptions. The aggregate example has $3{,}200$ Europe, Middle East, and Africa invoices, one per chunk. Even with all twenty returned invoices relevant and distinct, coverage is at most $20/3{,}200=0.625\%$. A sum over $0.625\%$ of the addends is wrong unless the omitted $3{,}180$ invoices cancel to zero. A routed design keeps the 620 lookups on retrieval. It sends the 180 aggregates to a text-to-SQL tool that scans all $3{,}200$ rows. That moves the failure surface from incomplete recall to query synthesis, which can be unit-tested against the database. It sends the 120 actions to a write API behind confirmation. Raising $k$ to cover every invoice would add $3{,}200 \times 50 = 160{,}000$ tokens. That does not fit a 128,000-token context window. At an assumed $\$3$ per million input tokens, the prefill would cost $\$0.48$ per query. The SQL aggregate returns one row of about 20 tokens. The retrieval route uses $8{,}000$ times as many input tokens for the same answer. Production analytics systems scan an index or columnar store for a complete sum rather than ranking rows by relevance.

Practical routing decisions

• If an answer depends on a few passages, retrieve.
• If it depends on every matching record, use a query or compute tool.
• If it changes the world, use an action tool behind confirmation.
• When $M \le k$, retrieval can still be complete. Listing three support tiers at $k = 20$ does not require SQL.
• Keep read-only tools inside retry loops and mutating tools outside them.
• A mutating call can enter the loop only when it is idempotent by key.
• Return passages plus source identifiers, not a pre-digested answer string.
• If latency requires server-side summarization, return both the summary and source identifiers.
• Add a router when more than one answer shape materially appears in traffic.
• Below roughly $10\%$ non-lookup volume, one retrieval path plus abstention can beat the cost of a router.
• Above that range, the $70\%$ ceiling in the worked traffic mix becomes material.
• An encoder-scale classifier is sufficient for triage in the source’s recommendation.
• Before adding RAG to an agent, ask whether an existing search tool already returns passages with source locations.
• If no SQL surface exists, use a metadata-filtered count only when the schema makes it complete. Otherwise abstain.
• A unified agent loop can keep citations only if its retrieval tool still returns passages with source identifiers. Measure citation coverage before and after the migration because the learned policy can skip retrieval.

1.5 When not to retrieve: the popularity curve and retrieval-induced harm

The measured failure

Retrieval is not a strict improvement. The cited PopQA study reports that retrieval flips a correct closed-book answer to a wrong answer on roughly $10\%$ of questions. The unassisted model wins outright on the high-popularity slice. Always retrieving can therefore improve the aggregate while regressing the most common queries. The failure is not necessarily fixed by a better reranker. It follows from the relationship between what the model already knows and what retrieval can spoil.

The popularity model

Let $s$ be subject popularity measured by monthly Wikipedia page views. Pre-training text follows a power-law distribution over entities. Popular subjects receive more supporting documents. Their facts are therefore more likely to survive in the weights. Let $p(s)$ be closed-book accuracy. Let $r(s)$ be retrieval-augmented accuracy. Define $g$ as the probability that the final RAG answer is correct conditional on the closed-book answer being wrong, and $h$ as the probability that the final RAG answer is wrong conditional on the closed-book answer being correct. Merely retrieving an answer-bearing passage is not the same as successful rescue. The following identity uses conditional rates for that slice; treating $g,h$ as constant across popularity is an additional scenario assumption. It does not imply that both accuracy curves must increase: with fixed rates, $r(s)=g+(1-g-h)p(s)$.

Retrieval helps when the rescue term exceeds the harm term. The odds form below assumes $h\gt 0$ and $p(s)\lt 1$. The threshold form assumes $g+h\gt 0$. If $g=h=0$, both policies tie everywhere; if $h=0\lt g$, retrieval cannot hurt in this model.

The equivalent accuracy threshold is below.

If $g$ and $h$ are constant across popularity and $p(s)$ is strictly increasing, the curves have at most one crossover; an actual crossing also requires the observed accuracy range to straddle $p^\star$. Flat segments can produce ties, and popularity-dependent $g(s)$ or $h(s)$ can create multiple crossings. The crossover is $s^\star$, where $p(s^\star) = p^\star$. Retrieval helps below $s^\star$ and harms above it. The threshold $p^\star$ is a property of the retriever and reader through $g / h$, not a property of one question. A small harm rate can dominate in the popular head. There, $1 - p$ is near zero and suppresses rescue, while $p$ is near one and preserves harm.

What creates the harm rate

On the slice where retrieval flips a correct answer, recall at 1 is significantly lower than overall recall. The first-ranked document is often topically related but does not contain the answer. A grounding prompt tells the reader to trust that supplied passage. The strongest distractor in the corpus can then override correct parametric knowledge. The harm rate is approximately rank-1 imprecision filtered through how strongly the prompt orders context use. Always-retrieve wins on average when $\mathbb E[(1-p)g]-\mathbb E[ph]\gt 0$. It still pays the loss on the popular head, which can carry most traffic and executive scrutiny. Never-retrieve fails on the rare tail. The source again cites the $10^{18}$ parameter extrapolation as the infeasible scale-only alternative. Popularity is not difficulty. For $g+h\gt 0$ and $h\gt 0$, the threshold satisfies $p^\star\lt 1$. A harmful region is possible near perfect closed-book accuracy, but it need not occur in a finite workload whose observed $p(s)$ stays below the threshold. Changing the generator, retriever or prompt can change $p,g,h$ together; the direction of a real crossover shift must be measured rather than inferred from only one changed quantity. A popular subject can still require fresh, ambiguous, or multi-step evidence. An obscure question can become easy once its one good page is present. The gate must estimate where the answer already lives, not whether the wording looks easy.

Worked example: 100,000 daily queries

The head has at least $10^{5}$ monthly page views and carries $55\%$ of queries. The torso spans $10^{3}$ to $10^{5}$ views and carries $30\%$. The tail is below $10^{3}$ views and carries $15\%$. Closed-book accuracies are $0.90$ for head, $0.55$ for torso, and $0.20$ for tail. The assumed rescue rate is $g = 0.55$. Traffic-weighted closed-book accuracy is $\bar p=0.690$. For this constructed example, assume a joint harmful-flip probability of $0.10$ across all queries and transfer a constant conditional harm rate across buckets. Then $h\bar p=0.10$, so $h=0.10/0.690\approx0.144928$ and $p^\star\approx0.791449$. This is a calibration assumption, not a measured bucket-level rate. If a reported $10\%$ flip rate is conditional on an initially correct answer, it means $h=0.10$ instead and does not justify this division. Under the stated joint-rate scenario, the head lies above the threshold. The torso and tail lie below the threshold. Keeping full precision in $h$, retrieved accuracies are approximately $0.824565$ for head, $0.717790$ for torso and $0.611014$ for tail. Closed-book accuracy is $69.00\%$ and always-retrieve accuracy is $76.05\%$. That is a $7.05$-percentage-point aggregate gain while the head loses approximately $7.5435$ percentage points. Selective retrieval keeps the $55\%$ head closed-book and retrieves for torso and tail. Its accuracy is approximately $80.1989\%$, beating always-retrieve by approximately $4.1489$ percentage points. The recovered probability mass is $0.55(0.90-0.824565217\ldots)\approx0.04148913$. The formerly stated equality $0.55\times0.075=0.041$ was not exact and rounded intermediate values too early. Each retrieved query adds five chunks of 400 tokens, or $2{,}000$ prompt tokens. Always-retrieve prefills 200 million extra tokens per day. Selective retrieval runs for $45{,}000$ queries and prefills 90 million extra tokens per day. That is a $55\%$ cut in prompt tokens, ANN searches, and reranker calls. The accuracy gain and cost cut point in the same direction because the removed work was harmful. The reported DPR top-$20$ retrieval accuracy near $78\%$ on Natural Questions is not a validation or bound for the assumed $g=0.55$. That benchmark measures a different target population and event; $g$ is end-to-end rescue conditional on a closed-book error in this workload. Measure conditional rescue and harmful-flip rates on matched queries before using the gate in production.

Practical gating decisions

• Slice evaluation into at least three popularity buckets before reporting an aggregate.
• For queries without linkable entities, use query frequency or head-phrase frequency in product logs as the proxy.
• Start with always-retrieve until the crossover is measured, then gate above it.
• If cost is the binding constraint, tune the gate to a spend target and report the accepted accuracy loss.
• Estimate $g$ and $h$ from matched closed-book/RAG outcomes on held-out queries; rank-$1$ precision and prompt trust are diagnostic covariates, not substitutes for those conditional rates.
• A stricter context-only prompt can change both $g$ and $h$; measure their net effect on $p^\star$ instead of assuming a fixed direction.
• Re-measure the crossover after a generator or retriever change.
• If a generator changes only $p(s)$ upward while $g,h$ remain fixed, the harmful region can expand; actual generator changes can also alter $g,h$.
• Holding other terms fixed, increasing $g$ or decreasing $h$ raises $p^\star$ and can shrink the harmful region; confirm that an improved retriever changes those outcome rates in the expected way.
• For a minor version change, the source recommends re-measuring on a 2,000-query sample.
• Alert on head-bucket accuracy separately because an aggregate cannot expose a majority-traffic regression.
• Keep the gate and re-measure quarterly. Retire it only when the harm region falls below an agreed traffic share.
• Without relevance labels, run both branches offline on a few thousand sampled queries. Score against resolved tickets, human feedback, or a model judge calibrated on held-out human judgments, then locate the bucket where closed-book stops losing.

Diagrams

The manifest records five figures and zero source tables. All five figures appear below. Figure $1.4$ is a numbered figure rendered as a Markdown matrix, not a source table.

Figure 1.1

Diagram description and source

flowchart TD
    Q[query x]
    D[D: non-parametric datastore. Capacity grows with stored rows. Write one embedded chunk and row. Reversible in seconds.]
    W[[θ: parametric weights. Capacity fixed before training. Write with a gradient pass over every parameter. Entangled and measured in GPU-hours.]]
    R[RET returns c_1 through c_k]
    G[GEN reads x and c_1 through c_k]
    Y[answer y]
    Q -->|retrieve| D
    Q -->|condition| W
    D --> R
    R --> G
    W --> G
    G --> Y
    style D stroke-dasharray: 5 5

Figure $1.1$: RAG is semi-parametric because it holds knowledge in two stores whose write costs differ by orders of magnitude: a row in D can be inserted, corrected, or deleted in seconds, while the same edit to $\theta$ costs a gradient pass and cannot be undone.

Figure 1.2

Diagram description and source

flowchart LR
    W[Knowledge lives in N weights. Diffuse. No address. Undated. Inseparable. Priced with fluency.]
    B1[1. Factual error. Frequency-weighted and lossy. Failed fix: scale toward 10^18 parameters.]
    B2[2. No verification. Tokens have no source return address. Failed fix: generated citations can be false.]
    B3[3. Update cost. One changed fact moves weights at 16N bytes. Failed fix: continual training causes forgetting.]
    B4[4. No opt-out. A tensor has no document delete. Failed fix: unlearning needs enumerated targets.]
    B5[5. Model size. C ≈ 6ND. Facts and fluency are priced together. Failed fix: the coupling is architectural.]
    W --> B1
    W --> B2
    W --> B3
    W --> B4
    W --> B5

Figure $1.2$: The five bottlenecks are five consequences of a single storage decision, which is why scaling the model relieves at most one of them and worsens two.

Figure 1.3

Diagram description and source

flowchart LR
    W[weights θ]
    D[datastore D]
    B1[Correctness: facts come from documents. Cost: every retrieval miss.]
    B2[Attribution: answers carry pointers. Cost: a pointer is a claim, not proof.]
    B3[Opt-out: documents can be deleted. Cost: weights must exclude them.]
    B4[Adaptability: new facts need no retraining. Cost: index freshness becomes an objective.]
    B5[Parameter efficiency: the generator can be smaller. Cost: index bytes and another latency stage.]
    W --> B1 --> D
    W --> B2 --> D
    W --> B3 --> D
    W --> B4 --> D
    W --> B5 --> D
    style D stroke-dasharray: 5 5

Figure $1.3$: Each RAG benefit is one responsibility moved from the weights to the datastore, and each move hands you a new failure surface on the receiving side.

Figure 1.4

Figure $1.4$: RAG occupies the one quadrant of the tool space where the call is safe to retry and its result is citable. Every other tool class surrenders one property or both, which is why attribution and recall-style evaluation transfer to retrieval and to nothing else.

Figure 1.5

accuracy
1.0 |                                      p(s) closed-book
    |                                 ____/
    |                         _______/#### retrieval-induced harm
    |          r(s) retrieval .......x........
0.5 |       ................../       | p > p⋆
    |                    ____/        |
    |        retrieval _/             |
    |        wins here                 | s⋆
0.0 +------------------------------------------------ popularity s
      10^1     10^2     10^3     10^4     10^5     10^6

production traffic by popularity bucket
tail 15%          | torso 30%                  | head 55%

Figure $1.5$: This is a schematic of a possible crossover, not a measured curve or a universal monotonic law. With fixed $g,h$ and strictly increasing $p(s)$, there is at most one crossing, and only if $p(s)$ straddles $p^\star=g/(g+h)$. In the constructed traffic mix, the harmful head is $55\%$ of traffic; its size and location must be measured for the actual product.

Whiteboard pack

What to draw

1. Draw a left box labeled D: addressable datastore and a right box labeled θ: entangled weights.
2. Put query x above both boxes.
3. Draw the retrieval arrow from $x$ to $D$ and label its output $c_1,\ldots,c_k$.
4. Draw the conditioning arrow from $x$ to the weights.
5. Join retrieved chunks and weights at a generator box, then draw answer y below it.
6. Write the five weight bottlenecks beside the weights: error, no source, update, no delete, coupled size.
7. Write the five datastore transfers beside $D$: correctness, attribution, opt-out, adaptability, efficiency.
8. Add a gate above retrieval and label it retrieve only when expected rescue exceeds expected harm.
9. Close with the routing rule: few passages use retrieval, complete sets use query tools, and state changes use confirmed action tools.

Spoken script

The design has two knowledge stores. The weights hold behavior and compressed prior knowledge, while the datastore holds addressable facts. A query can go straight to the generator or retrieve passages first. This split fixes five weight bottlenecks: long-tail error, missing provenance, expensive updates, no bounded deletion, and facts priced with fluency. Each fix creates a datastore cost, including misses, stale indexes, index memory, and extra prefill. I gate retrieval because a distracting passage can override a correct answer. I also route aggregates to complete query tools and state changes to confirmed action tools.

Interview traps

1. Why not fine-tune on every company document?

Fine-tuning can improve fixed-corpus factual fit, but it does not create source addresses or bounded deletion. Price the proposal with $16N$ memory and $6ND$ training work, then keep changing, attributable, or deletable facts in the datastore.

2. Can a larger datastore justify a much smaller generator?

Not automatically. A larger store increases near-duplicates, contradictions, and reading load, so compare grounded accuracy and conflict resolution at fixed $k$ before shrinking the generator.

3. Why can recall at 20 be high while totals are wrong?

Recall asks whether relevant passages appear, but an aggregate needs all $M$ matching records. At $M = 3{,}200$ and $k = 20$, coverage is capped at $0.625\%$, so use SQL or another complete compute tool.

4. When should retrieval not be used?

Do not use retrieval for complete aggregates, mutating actions, or popular-subject questions above the measured crossover where harm exceeds rescue. Also skip it when the model already answers reliably and the added passage is more likely to distract than help.

5. Does a citation prove that an answer is supported?

No. Citation coverage only proves that a marker exists, so measure sentence-to-span support with an entailment check and budget verified sample-and-rerank by its generation multiplier.

Key numbers

00-START-HERE · 01-READING-ROUTES · 00_09_about_the_author · 02_the_competing_answers_to_the_same_problem