Chapter 9: Prompting for Retrieval-Augmented Systems

00-START-HERE · 01-READING-ROUTES · 08_reading_the_machine_circuits_induction_heads_and_attribution · 10_prompt_sensitivity Purpose: Explain where prompting ends, where retrieval begins, how reasoning scaffolds change decoding, and why more calls or more passages can reduce answer quality.

TL;DR

• A Retrieval-Augmented Generation (RAG) prompt has four cumulative rungs: completion, instruction, In-Context Learning (ICL), and query-time retrieval.
• The prompt hierarchy distinguishes illustrative examples from source evidence; both can be query-dependent. Demonstrations may be fixed or query-selected; retrieved evidence is selected for the query. The distinction is illustrative role and evidence provenance, not a universal fixed-versus-dynamic boundary.
• Chain-of-Thought (CoT), self-consistency, Tree of Thoughts, and Graph of Thoughts change decoding. They do not change the weights or the retrieved passages.
• Self-consistency estimates an answer marginal by sampling paths and counting final answers. It reduces variance, but it cannot correct bias from a wrong retrieved context.
• A scaffold is a program the model must execute. An all-required-stages scenario discounts accuracy by conditional stage reliability; exploratory branches may tolerate failures.
• Raising top-$k$ can improve recall and reduce answer accuracy. More passages increase competition for the generator’s attention.
• Choose prompt rungs, scaffold depth, and top-$k$ with measured failure partitions, execution reliability, ranker margin, latency, and cost.

The story

Picture an open-book exam with four rungs on a ladder beside the student’s desk. On the first rung, the student gets only the question and continues the text from memory. This is completion, which means predicting what text comes next without special directions or external material. On the second rung, the examiner adds an instruction. The student now knows to answer instead of continuing the question. The instruction changes behavior, but it does not add a missing policy fact. On the third rung, the examiner puts worked examples on the board before seeing the new question. This is In-Context Learning (ICL), which means guiding the current answer with examples while leaving permanent training unchanged. The examples teach answer form, citation style, refusal wording, or a JavaScript Object Notation (JSON) envelope, but they cannot carry every changing fact in a large corpus. On the fourth rung, the examiner lets the student find the relevant page after reading the question. This is Retrieval-Augmented Generation (RAG), which means finding evidence after the request and giving it to the answer writer. The page can be fresh, query-specific, deletable, and citable, but the student must still find and use it correctly. Now give the student scratch paper. A single reasoning chain is one worked path, while a fan of five paths lets the student solve the problem five ways and count the final answers. A tree adds branching, scoring, pruning, and continuation, while a graph can also merge or revise partial work. Each extra move asks the same student to execute another instruction. If every move is $98\%$ reliable, thirty moves succeed together only $54.5\%$ of the time. Too much scaffolding, meaning added scratch-work procedure, turns the exam ladder into a maze. The same warning applies to the open book. Five pages may include the right page and few distractors, while forty pages can raise recall, meaning how often the right page is present, and bury the answer among plausible competitors. A better ranker gives the right page a larger score margin, and that margin decides how many pages the student can safely receive. The interview lesson is concrete. Do not fix a missing page by rewriting instructions, biased pages by sampling more answers, or weak ranking by buying a longer desk. Climb the right rung, add only executable scratch work, and send only as many pages as the student can reliably distinguish.

Decoder table

Core mechanics

Mathematical conventions

Natural logarithms measure likelihood in nats. The $8$-billion-parameter examples use $W\approx2NT$ at assumed $P=3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$ and $\$2.50$ per GPU-hour. These are parameter-work equivalents, not measured GPU service time, end-to-end latency or p95 guarantees; attention, bandwidth, queuing, orchestration and cache management are omitted. Prefix sharing avoids repeated prefix computation but not branch-specific attention or KV residency. Probabilities below are scenario inputs unless explicitly identified as reported empirical results. Percentage-point differences are absolute, not relative gains. Preserve precision until display.

9.1 The prompt hierarchy: completion to instruction to ICL to RAG

The four-slot input

The slots are instruction, demonstrations, retrieved passages, and query.

Rung 1: completion

Rung 2: instruction

Wei et al. (2022) built FLAN over 62 datasets in 12 task clusters. Their instruction-tuned 137-billion-parameter model beat zero-shot GPT-3 175 billion on 20 of 25 held-out datasets. Longpre et al. (2023) scaled the recipe to $1{,}836$ tasks from 473 datasets. Ouyang et al. (2022) added preference optimization.

Rung 3: in-context learning

Brown et al. (2020) made this paradigm a headline result of GPT-3.

Rung 4: retrieval-augmented generation

Mechanically, rung 4 enters through the same context channel as rung 3. The transformer has no native slot type that distinguishes a demonstration from a passage. Query dependence is not exclusive to RAG: demonstration selection can also depend on the live input. Here demonstrations illustrate task behavior, while retrieved passages supply source evidence.

Capacity

Binding latency

Why the growing system prompt loses

The rungs stack

Worked policy assistant

The system uses an 8-billion-parameter generator and serves $10{,}000$ queries per day. Its window is $8{,}192$ tokens. The corpus has $4{,}000$ policy documents averaging 250 tokens. Serving uses $1.6 \times 10^{10}$ FLOPs per token at $3.4 \times 10^{14}$ FLOPs per second. Graphics processing unit (GPU) time costs $\$2.50$ per hour. The question uses 40 tokens. Configuration 1 uses a 120-token instruction plus the question. It totals 160 tokens, $2.56 \times 10^{12}$ FLOPs, and $7.5\,\mathrm{ms}$ of prefill. Configuration 2 adds four demonstrations of 250 tokens each. The 1,000-token block costs $1.6 \times 10^{13}$ FLOPs and $47\,\mathrm{ms}$. Configuration 3 adds eight retrieved chunks of 400 tokens each. The 3,200-token block costs $5.12 \times 10^{13}$ FLOPs and $151\,\mathrm{ms}$. The complete 4,360-token prompt costs $6.98 \times 10^{13}$ FLOPs and $205\,\mathrm{ms}$ of prefill. At $\$2.50$ per GPU-hour, it costs $\$1.42 \times 10^{-4}$ per query, $\$0.14$ per thousand queries, and $\$1.42$ per day. The retrieval block alone costs $\$0.10$ per thousand queries. Lewis et al. (2020) reported $44.5$ exact match on Natural Questions for RAG-Sequence. It used BART-large with roughly 400 million parameters and a Wikipedia index. Closed-book T5-11B scored $34.5$ with a model 27 times larger. Retrieval bought ten exact-match points that $10.6$ billion additional parameters did not buy.

Operational decisions from the hierarchy

• Route content plus an absent supporting passage to retrieval.
• Route content plus a present supporting passage to generation or ordering.
• Route form failures to instruction or demonstrations.
• Use an instruction-tuned checkpoint for RAG generation.
• Use a base checkpoint only when the undistorted token distribution is the goal, such as likelihood ranking or infilling.
• Start with zero demonstrations.
• Add demonstrations only when measured format gains justify displacing retrieved chunks.
• Two format examples can be worthwhile if they cut a $30\%$ JSON failure rate to near zero.
• Put facts in retrieval and behavior in instructions or demonstrations.
• Move facts into weights only when they change slowly, apply universally, and never need citation or deletion.
• Move facts out of weights when staleness approaches the change interval, erasure is required, or an auditor needs provenance.
• For strict traceability, require cited spans, entailment checks, abstention without support, and an unsupported-claim metric. Enforce evidence for asserted claims rather than claiming that parametric knowledge is unused.

9.2 Reasoning scaffolds: CoT, self-consistency, Tree of Thoughts, and Graph of Thoughts

What a scaffold changes

In the motivating support assistant, the supporting passage appears in the assembled context on $94\%$ of the evaluation set. The hard questions require information from more than one chunk. A short prompt, with token count depending on the tokenizer, such as Let's think step by step competes with a thirty-call tree under an $800\,\mathrm{ms}$ 95th-percentile latency budget.

Chain-of-thought as a marginal over paths

Three complete paths can reach the answer “yes, with a $\$40$ cap” with joint probabilities $0.20$, $0.18$, and $0.15$. Their answer mass totals $0.53$. One path can reach “no” with probability $0.25$. If the $0.25$ path is the largest complete-path probability and all unlisted paths are smaller, maximum-probability-path decoding selects “no” while the stated answer marginal favors “yes.” The listed probabilities alone do not determine token-greedy behavior, and the remaining $0.22$ probability mass must be allocated consistently to complete the example. Kojima et al. (2022) used Let's think step by step plus a second answer-extraction call. On GSM8K, text-davinci-002 rose from $10.4\%$ to $40.7\%$.

Self-consistency

At $p = 0.53$ and $m = 5$, the vote-share standard error is $0.223$. At $m = 40$, it is $0.079$. Self-consistency uses a counter, not a scorer. That separates it from best-of-$N$ against a reward model.

Majority-vote arithmetic

Voting can improve accuracy when the correct answer has sufficient probability under the sampled distribution. It does not generally eliminate systematic bias; treating vote share as calibrated confidence requires validation.

Chain, fan, tree, and graph

Graph of Thoughts permits aggregation and refinement edges that a tree cannot draw. Yao et al. (2023) reported GPT-4 on Game of 24 at $4\%$ with CoT. Self-consistency over 100 samples scored $9\%$. A breadth-5 tree scored $74\%$. Besta et al. (2024) reported Graph of Thoughts improving sorting quality by $62\%$ over Tree of Thoughts at more than $31\%$ lower cost. Game of 24 provides an exact arithmetic state evaluator. A half-finished policy answer does not. The 70-point gain measures the evaluator as much as the tree. Wei et al. (2022) found CoT gain to be an emergent property of scale in their setting. Below roughly 100 billion parameters, their models produced fluent but illogical chains and did not beat standard prompting. Distillation on reasoning traces moved the threshold downward. The remaining failure is unchanged. A scaffold the generator cannot execute costs tokens and buys nothing.

Worked scaffold costs

The shared prompt has $120$ instruction, $8\times400=3{,}200$ evidence, and $40$ question tokens: $3{,}360$ input tokens. Each chain uses $250$ output tokens. Assume each sampled chain has correctness probability $0.60$ and the two-answer IID voting model; this is not automatically the accuracy of a greedy chain under another decoding policy. One greedy chain processes $3{,}610$ tokens. It costs $5.78 \times 10^{13}$ FLOPs, $170\,\mathrm{ms}$ of GPU time, and $\$1.18 \times 10^{-4}$ per query. Five uncached self-consistency samples process $18{,}050$ tokens. They cost $2.89 \times 10^{14}$ FLOPs, $849\,\mathrm{ms}$, and $\$5.90 \times 10^{-4}$. Accuracy is $68.3\%$. Five samples with a shared key-value (KV) cache process $4{,}610$ tokens. They cost $7.38 \times 10^{13}$ FLOPs, $217\,\mathrm{ms}$, and $\$1.51 \times 10^{-4}$. They preserve the same accuracy and are $3.9$ times cheaper. The samples can batch, so $217\,\mathrm{ms}$ is GPU time rather than wall-clock time. A tree with $b = 3$, $d = 3$, and $k = 2$ generates 3, then 6, then 6 children. Fifteen generation calls plus fifteen score calls make thirty model calls. Each generation uses 120 state tokens plus 80 generated tokens. Each score uses 120 state tokens plus 5 generated tokens. The cached tree adds $4{,}875$ marginal tokens to one 3,360-token prefill. It processes $8{,}235$ tokens, $1.32 \times 10^{14}$ FLOPs, $388\,\mathrm{ms}$, and $\$2.69 \times 10^{-4}$. Without prefix reuse, the $30$ tree calls reread $30\times3{,}360$ prompt tokens and add $15(200+125)=4{,}875$ marginal tokens, totaling $105{,}675$ processed tokens. The proxy is $1.6908\times10^{15}\,\mathrm{FLOP}$, approximately $4.973\,\mathrm s$ and $\$0.0034534$. This is $105{,}675/8{,}235\approx12.83$ times the cached tree and $105{,}675/3{,}610\approx29.27$ times a single-chain token-work proxy. Three levels each contain a generation stage and a dependent scoring stage; parallel siblings do not remove these dependencies. Wang et al. reported PaLM-540B on GSM8K at $56.5\%$ for greedy CoT and $74.4\%$ with $40$ sampled chains. Plugging $p=0.565$ into an IID two-answer majority model with random tie-breaking predicts approximately $79.4052\%$, not $74.9\%$. More importantly, greedy accuracy does not establish each sampled chain's correctness probability, and real answers need not have two labels. The reported result therefore does not validate that toy majority calculation.

Operational decisions for scaffolds

• Partition failures before choosing a scaffold.
• Use scaffolds only where the supporting passage is present.
• Start with CoT plus self-consistency at $m = 5$ and a shared prefix cache.
• Raise $m$ toward 20 when three-two splits routinely decide answers.
• Drop to one greedy chain when the task is extraction and all five samples agree on the evaluation set.
• Use sampling temperature around $0.7$ for an ensemble and greedy decoding for one answer.
• Deterministic greedy decoding with fixed inputs and serving state returns the same output; nondeterministic kernels or other stochastic components can still vary.
• Lower temperature when parseable output rate falls. Do not infer that raising temperature fixes a unanimously wrong system; diagnose missing evidence and systematic bias first.
• Reject a tree or graph until its evaluator beats chance.
• Exact checks, Structured Query Language (SQL) parsing, arithmetic checks, and entailment against a cited span can qualify.
• Budget a fan as one parallel round and a depth-$d$ tree as $d$ serial generate-and-score rounds.
• Log the full vote distribution. A three-two split and a five-zero split should not produce the same confidence signal.

9.3 When more scaffolding makes things worse

The execution discount

In the source example, that pipeline scores $61.2\%$ on the full multi-hop slice. A two-line fan of five scores $68.0\%$ in one round at one sixth of the calls. For $c$ stages that are all required to succeed, let $q_j$ be the conditional probability that stage $j$ succeeds given previous required successes. The chain rule then gives the product below without an independence assumption. Using unconditional $q_j$ would need independence or another justified joint model. Requiring every exploratory call to succeed is a pessimistic scenario, not a necessity for all tree algorithms.

Realized accuracy mixes the perfectly executed design with its misfire state:

For $p_s\gt p_d$, the equality with incumbent accuracy is $e^\star=(A_0-p_d)/(p_s-p_d)$. A target above $p_s$ is infeasible, a target at or below $p_d$ requires no execution improvement, and $p_s=p_d$ has no meaningful execution crossover. An architectural label does not identify these conditional accuracies.

Use $A_0 = 0.683$, $p_s = 0.85$, and $p_d = 0.45$. The equality is $e^\star=(0.683-0.45)/(0.85-0.45)=0.5825$. A thirty-call tree therefore needs per-call reliability of $0.982$. At $0.98$ per call, the tree executes end to end only $0.545$ of the time. The design gain is additive, but the execution discount is geometric.

Deterministic aggregation and reversible decisions

Jiang et al. (2023) built the large language model (LLM) system LLM-Blender. It ranks outputs from eleven open-source models pairwise and sends top candidates to a fusion model. Peiyi Wang and colleagues (2023) found that an LLM judge can change its verdict when candidate order is swapped. Their fix scores both orders and averages, which doubles comparison calls. Self-review can reinforce the model’s own answer and raise confidence without improving correctness. The source connects this mechanism to model collapse.

Reliability audit

A five-sample fan gets all five parseable outputs with probability $0.904$ at $98\%$ per-call parseability. Exactly four parse with probability $0.092$. Fewer than four parse with probability $0.004$. A four-sample vote with random tie-breaking scores $0.648$ at $p = 0.60$. Define the parseability policy explicitly: $J\sim\mathop{\mathrm{Binom}}\nolimits(5,0.98)$ valid samples, correctness $p=0.60$ conditional on validity, random tie-breaking for even $J$, and fallback accuracy $0.45$ when $J=0$. Summing over $J$ gives accuracy $0.67923578$, or $67.9236\%$, compared with ideal five-vote accuracy $0.68256$. Other fallback, invalidity/correctness dependence or retry policies change the result. A thirty-call tree at $98\%$ execution and a perfect scorer realizes $0.668$. The scenario tree uses $30$ calls versus $5$ for the fan and has three dependent generate-and-score levels. At $q=0.98$ and perfect pruning, its accuracy $0.66819373$ trails the explicitly modeled fan by $1.1042$ percentage points; arithmetic work is not a wall-clock latency guarantee. If its evaluator keeps the correct branch $90\%$ of the time, full execution falls to $0.398$. Realized accuracy falls to $0.609$, seven points below the fan. Under this all-$30$-calls-required scenario, pruning survival $r$ per level would need $r^3\ge(A_{\mathrm{fan}}-0.45)/(0.40\times0.98^{30})$, hence $r\ge1.016592\gt 1$. No feasible pruning reliability can make this tree beat that fan while the other assumptions stay fixed. The earlier $83.5\%$ threshold was incorrect. Improve execution, design/fallback accuracy or the all-calls-required design before considering evaluator gains. Measure this on 200 labeled pruning decisions before writing orchestration. On Game of 24, a perfect finite search has $p_s = 1.0$ and the self-consistency fallback is $0.09$. Within the specified mixture with $p_s=1$ and $p_d=0.09$, the observed $0.74$ can be represented by $e=(0.74-0.09)/(1-0.09)=5/7\approx0.714286$. This is an implied mixture coefficient, not an independently measured execution rate or proof that all remaining error is execution failure. Even a free exact evaluator leaves a large execution discount. Count model-executed steps as a reliability budget. Prefer fewer than roughly ten unless the weakest stage has a measured $q_j$. Log stage validity, correct-branch survival, truncation, and dropped constraints separately from final answer accuracy. Ablate downward one stage at a time. On 500 questions at accuracy near $0.68$, one arm has standard error about $0.021$. A two-point change on $500$ questions is not resolved by looking at one arm's standard error alone; use the paired outcomes or an independent difference interval as appropriate. With roughly $100$ discordant pairs and a net difference near $15$, the uncorrected signed McNemar normal statistic is approximately $z=15/\sqrt{100}=1.5$; the corresponding chi-square statistic is $2.25$, below $3.841$ for a nominal two-sided $5\%$ test. Exact counts, discreteness and continuity correction matter.

9.4 Retrieving more can hallucinate more

Recall is not answer accuracy

The naive model assumes perfect use of the gold passage and closed-book fallback otherwise.

This expression can only rise with recall. Production breaks its two assumptions. When the gold passage is absent, the model can ground in a plausible wrong passage instead of returning to parametric memory. This co-occurrence substitution gives no-gold accuracy $m$, with $m \lt p_0$.

Utilization and margin

The relative penalty per added passage is $\beta = 1 - \Phi(\mu)$. At $\mu = 2.0$, $\beta = 0.0228$. At $\mu = 2.5$, $\beta = 0.0062$. At $\mu = 3.0$, $\beta = 0.00135$. A reranker can change selected passages and order, but a larger retrieval-score margin does not automatically increase the generator’s effective score margin. That causal link requires a separate intervention or measurement.

The non-monotone accuracy model

$G(k)=R(k)u(k)$ is the probability of selecting the gold passage in this scenario, not measured grounding or faithfulness. The accuracy formula further assumes correctness whenever gold is selected and constant fallback accuracy $m$ in every other state, including gold present but not selected. Measuring $m$ only after deleting gold does not establish that second conditional accuracy. Recall rises with $k$ while utilization falls. Karpukhin et al. (2020) reported Dense Passage Retrieval (DPR) accuracy on Natural Questions of $78.4\%$ at top-20 and $85.4\%$ at top-100. The reported anchors bracket $20\le k\le100$. Use a log-linear scenario curve there; values at $5\le k\lt 20$ are extrapolations, not interpolation or reported DPR measurements:

Marginal recall decays as $\frac{c}{k}$. Marginal dilution stays near constant at $\beta$. The optimum solves:

At $\mu=3$, the continuous stationary point of this scenario is $k\approx39.571$. Check nearby integers and feasible-domain endpoints. This remains a toy optimum, not a deployment guarantee. At $\mu=2.5$, the continuous stationary point of this scenario is $k\approx9.301$. Check nearby integers and feasible-domain endpoints. This is below the $20$–$100$ anchor bracket and is an extrapolation, not a validated optimal retrieval depth. At $\mu=2$, the continuous stationary point of this scenario is $k\approx2.711$. Check nearby integers and feasible-domain endpoints. This is below the $20$–$100$ anchor bracket and is an extrapolation, not a validated optimal retrieval depth. A stationary point below the measured range calls for a held-out depth sweep and ranker/utilization diagnosis; it does not mathematically mandate a particular repair. Half a standard deviation of margin moves the optimum by more than a factor of four. A longer context window does not change the number of competing passages. Liu et al. (2023) found that extended-context variants of the same models were no better at using their context and barely moved the accuracy curve.

Worked retrieval audit

The chunks use 512 tokens. Closed-book accuracy is $0.42$. Misgrounded accuracy after deleting every gold passage is $0.31$. Input costs $\$3$ per million tokens. Under the stated scenario at $k=5$ and $\mu=2.5$, $R(k)\approx0.723705$, $u(k)\approx0.975392$, the gold-selection proxy is $0.705896$, and $A(k)\approx0.797068$. These are derived values, not measured bi-encoder or reranker outcomes. Grounding is $0.706$ and answer accuracy is $0.797$. The 2,560-token prompt costs $\$0.0077$ per query. Under the stated scenario at $k=40$ and $\mu=2.5$, $R(k)\approx0.814147$, $u(k)\approx0.784325$, the gold-selection proxy is $0.638556$, and $A(k)\approx0.750604$. These are derived values, not measured bi-encoder or reranker outcomes. Utilization falls to $0.784$. Grounding falls to $0.638$ and accuracy falls to $0.751$. The 20,480-token prompt costs $\$0.0614$ per query. The input bill rises eightfold while scenario accuracy falls by $4.6465$ percentage points. As a separate counterfactual, increase the generator-selection parameter from $\mu=2.5$ to $\mu=3$ while holding the scenario recall curve fixed. A top-$100$ cross-encoder does not establish those assumptions by itself: reranking also changes the recall curve of the presented subset, and generator utilization must be measured. Under the stated scenario at $k=40$ and $\mu=3$, $R(k)\approx0.814147$, $u(k)\approx0.948682$, the gold-selection proxy is $0.772367$, and $A(k)\approx0.842933$. These are derived values, not measured bi-encoder or reranker outcomes. Only the toy utilization parameter changes in that calculation. If the actual passages, order and generator computation are unchanged, altering an external score alone cannot change the answer. Under the stated scenario at $k=10$ and $\mu=3$, $R(k)\approx0.753853$, $u(k)\approx0.987916$, the gold-selection proxy is $0.744743$, and $A(k)\approx0.823873$. These are derived values, not measured bi-encoder or reranker outcomes. It uses $5{,}120$ tokens and costs $\$0.0154$. The last $1.9$ points from 10 to 40 passages cost four times the input bill. The input-only difference is $(40-10)\times512\times\$3=\$46{,}080$ per million queries, before the additional $15{,}360$ prompt tokens of prefill work. The modeled accuracy gain is conditional on the same scenario recall and utilization assumptions. Under the stated scenario at $k=100$ and $\mu=2$, $R(k)\approx0.854000$, $u(k)\approx0.102460$, the gold-selection proxy is $0.087501$, and $A(k)\approx0.370376$. These are derived values, not measured bi-encoder or reranker outcomes. The $\mu=2$ scenario crosses the $0.42$ baseline at continuous $k\approx73.238$; the first integer below baseline must be checked separately. Liu et al. (2023) observed the inversion at 20 documents in multi-document question answering. Omitted positional effects can change the optimum, but do not make this stationary point a proven upper bound. Correlated distractors, multiple gold passages, answer-dependent fallback and changes in reranked recall alter the curve. Duplicate correlation can reduce the effective number of independent competitors without guaranteeing that deduplication improves answer accuracy. Gate releases on answer accuracy, not recall at top-$k$. Measure $m$ by deleting the gold passage from every context. Measure dilution by fixing the gold position and injecting distractors. Prefer a score cliff over a fixed top-$k$ when scores are calibrated. Retrieve conditionally when closed-book answers are reliable. Retrieve unconditionally when every answer requires a citation, while accounting for the accuracy trade. Mallen et al. (2023) found the unassisted model beating retrieval augmentation on high-popularity subjects.

Diagrams

Figure 9.1

Diagram description and source

flowchart BT
    C["1. completion: bound at pre-training, theta_pre, continues q, +0 tokens"]
    I["2. instruction: bound at post-training, theta_IT, command, +120 tokens, 5.6 ms"]
    L["3. in-context learning: bound at prompt deploy, fixed before q, +1,000 tokens, 47 ms"]
    R["4. RAG: bound per request, passages selected after q, +3,200 tokens, 151 ms"]
    C --> I
    I --> L
    L --> R

Figure $9.1$: The four rungs are cumulative rather than alternative, and they differ in the one property that decides everything else - how late the information they carry can be bound, from the pre-training corpus up to the request itself - with the price of that lateness paid in prompt tokens on every query.

Figure 9.2

Diagram description and source

flowchart LR
    subgraph Chain["chain - CoT: 1 call"]
        q1[q] --> c1((step)) --> c2((step)) --> a1[a]
    end
    subgraph Fan["fan - self-consistency: m parallel calls"]
        q2[q] --> f1((path 1)) --> A1[A]
        q2 --> f2((path 2)) --> A2[A]
        q2 --> f3((path 3)) --> B1[B]
        A1 --> count[count]
        A2 --> count
        B1 --> count
    end
    subgraph Tree["tree - Tree of Thoughts: about dkb calls in d serial rounds"]
        q3[q] --> t1[0.9]
        q3 --> t2[0.2]
        q3 --> t3[0.7]
        t1 --> t11((child))
        t1 --> t12((child))
        t3 --> t31((child))
        t3 --> t32((child))
        t12 --> a3[a]
    end
    subgraph Graph["graph - Graph of Thoughts: merge and refine"]
        q4[q] --> g1((thought 1))
        q4 --> g2((thought 2))
        g1 --> merge((merge))
        g2 --> merge
        merge --> a4[a]
        merge --> refine((refine))
        refine --> merge
    end

Figure $9.2$: The four scaffolds differ in one structural property - what an edge is allowed to do - and every cost follows from it: the fan is embarrassingly parallel and needs no scorer, while the tree buys exploration only by adding a serial round per level and a state evaluator at every node.

Figure 9.3

realized accuracy
90% |                                             tree reaches 85%
80% |                                        ____/
70% |---------------- fan of five: 68.3% ---*      break-even q = 0.982
60% |                              _________/
50% |             _______________/
40% |____________/
     0.90          0.94          0.98        1.00
              per-call execution reliability q

tree: b = 3, d = 3, 30 model calls
A = 0.45 + 0.40 q^30

Figure $9.3$: The scenario curve $A=0.45+0.40q^{30}$ assumes every one of $30$ calls must succeed with common conditional reliability $q$. It crosses an ideal $0.683$ fan target at $q=(0.5825)^{1/30}\approx0.98215$. At $q=0.98$, even perfect pruning remains below the parseability-modeled fan, so no $r\le1$ rescues the tree under the unchanged assumptions. This is a reliability illustration, not an empirical latency or accuracy frontier.

Figure 9.4

answer accuracy A(k)
90% |
80% |  mu = 3.0 _________* k = 40 ______
70% |  mu = 2.5 * k = 9  \_______________
60% |                      \_______________
50% |  mu = 2.0 \__________________________
42% |----------- closed book --------------* k about 73
40% |                                      \
     5         10        20        40      100
             passages sent to generator k, log scale

m = 0.31
R uses DPR top-20 recall 0.784 and top-100 recall 0.854

Figure $9.4$: The curve $A(k)=m+(1-m)R(k)\Phi(\mu)^{k-1}$ is an explicitly conditional toy model: fixed gold score, independent Gaussian distractors, perfect correctness on gold selection and common fallback elsewhere. Only recall at $20$ and $100$ is anchored to reported DPR results. Values below $20$, generator margins and the displayed optima are constructed, not measured. A reranker must demonstrate its own presented-set recall and utilization curve.

Whiteboard pack

What to draw

1. Draw four stacked boxes labeled completion, instruction, ICL, and RAG.
2. Write the binding time beside each box.
3. Add the four prompt slots: instruction, demonstrations, retrieved passages, query.
4. Draw a fan of five sampled chains with a counter.
5. Draw a depth-3 tree with generation and score calls.
6. Write $e = \prod_{j=1}^{c} q_j$ under the tree.
7. Draw recall rising with top-$k$.
8. Draw utilization falling with top-$k$.
9. Multiply them into a curve with an interior maximum.

Spoken script

Start with the four prompt rungs. Completion recalls from weights, instruction sets behavior, examples fix form, and retrieval binds evidence after the query. Then separate retrieval from decoding. A reasoning scaffold only changes how many paths we sample and how we combine them. A fan counts answers, while a tree needs serial scoring and reliable pruning. Its execution probability multiplies across calls, so thirty $98\%$ reliable calls succeed together only $54.5\%$ of the time. Finally, split retrieval quality into recall and utilization. Raising top-$k$ improves recall but adds distractors. Ranker margin decides where answer accuracy peaks.

Interview traps

1. Is RAG just in-context learning with a vector database?

Mechanically, both condition frozen generator weights on context tokens. Demonstrations can be query-selected just as evidence can; distinguish their role and provenance rather than claiming a universal binding-time difference. A vector database is one retrieval implementation, not the definition of RAG.

2. Will self-consistency turn 60% grounded question-answering accuracy into 75%?

No. Majority-of-five reaches $68.3\%$, while majority-of-nine reaches $73.3\%$. Voting can recover probability mass already present across independent paths, but it cannot correct a wrong or missing passage that biases every sample.

3. When does more scaffolding make results worse?

It gets worse when the scaffold adds many model-executed stages, weak evaluators, irreversible pruning, or systematic comparators. Thirty calls at $98\%$ per-call reliability execute together only $54.5\%$ of the time. A chance scorer that keeps 2 of 3 branches preserves the correct path through three levels only $29.6\%$ of the time.

4. When does more retrieval make results worse?

It gets worse when marginal recall from another passage is smaller than the utilization loss from another competitor. At ranker margin $2.5$, moving from 5 to 40 passages raises recall from $0.724$ to $0.814$ but lowers answer accuracy from $0.797$ to $0.751$. Measure distractor sensitivity and no-gold accuracy before raising top-$k$.

5. Does a million-token window remove the need for ranking?

No. A larger window removes a token-capacity limit but does not remove competition among passages. Dropping the reranker lowers the gold passage’s margin while raising the number of competitors, which you can test by fixing the gold position and sweeping distractor count.

Key numbers

00-START-HERE · 01-READING-ROUTES · 08_reading_the_machine_circuits_induction_heads_and_attribution · 10_prompt_sensitivity