Chapter 26: Iterative, Recursive, and Agentic Retrieval

00-START-HERE · 01-READING-ROUTES · 25_routing_and_adaptive_retrieval · 27_fine_tuning_the_generator Purpose: Explain why a single retrieval call fails when the useful query appears only after reasoning begins, then compare the retrieval-augmented generation (RAG) control loops that retrieve again, critique evidence, learn search behavior, fan out across sources, and stop before extra rounds reduce value.

TL;DR

• One-shot retrieval fixes its query and context before generation. Hidden bridges may need another search; unresolved references can instead need a history-aware rewrite, and stale evidence needs a currency-aware check.
• ITER-RETGEN feeds a draft into the next retrieval query and replaces the retrieved set. With fixed chunk lengths, each generator prompt stays bounded and cumulative full-replay prompt work grows linearly with rounds.
• Recursive methods derive the next hop from partial reasoning. IRCoT accumulates passages and reasoning; Self-Ask retains short question-answer history, trading evidence detail for a smaller context.
• FLARE tests ordinary draft-token probabilities. Accept a non-triggering draft; on a trigger, mask uncertain spans, retrieve and regenerate. Confidence is not a factuality certificate.
• CRAG uses a separately trained evaluator and two thresholds. The ambiguous band can preserve useful local evidence while adding web results, but does not guarantee zero answer harm.
• Self-RAG learns reflection tokens. Ranking $K$ candidates costs candidate decoding; low reflection-token overhead does not make the branches free or their critiques independent.
• Search-R1 trains policy-generated reasoning, searches and answers while treating retrieved information as context, not authored actions. GRPO removes a learned critic but needs sampled groups; identical rewards give zero reward advantages, not necessarily zero total gradient.
• Agent fan-out can overlap source-specific work or reduce context-attention work, at the price of duplicated prompts, summaries and coordination. Stop using a validated marginal-value model, not confidence alone; keep hard resource and safety caps.

The story

Imagine a field team crossing a chain of islands with one radio call to headquarters. The first island is easy to name from the mission brief. The second is reachable only after the team reads a sign on the first. A dispatcher with one frozen query may not name the next island before seeing its sign. ITER-RETGEN lets the scout radio back a draft route; headquarters joins that draft to the original mission and sends a replacement map. A map index that already links the islands could sometimes find both at once, so the analogy is a failure mode, not an impossibility theorem. IRCoT keeps maps and reasoning in a growing field notebook. Self-Ask retains short questions and intermediate answers instead of every map. Its notes are lighter, but they still grow and may omit qualifiers or provenance needed later. FLARE sends a scout ahead and masks uncertain guesses before requesting another map. CRAG adds a separately trained inspector who filters useful maps, replaces bad ones, and supplements ambiguous ones. Separate training does not make inspector errors statistically independent of the scout’s errors. Self-RAG gives the scout compact learned signal flags, but sending a scout down every candidate trail still costs work. Search-R1 lets terminal reward shape policy actions. If every expedition receives the same reward, GRPO’s reward-advantage signal vanishes; regularization or other loss terms may still update the policy. Agentic retrieval sends separate teams to separate sources. Parallel follow-ups can help, and splitting large contexts can also help before any follow-up; summaries and coordination can erase those gains. Stop when expected evidence benefit no longer compensates for dilution and resource cost, with uncertainty and hard caps stated.

Decoder table

Core mechanics

Mathematical conventions

Use natural logarithms, decimal storage units and full precision until rounding: $1\,\mathrm{GB}=10^9\,\mathrm B$ and $1\,\mathrm{TB}=10^{12}\,\mathrm B$. Probabilities and accuracy are fractions; an absolute increase of $0.01$ is one percentage point, not necessarily a one-percent relative increase. Products of hop probabilities below require either explicitly conditional factors or a stated independence model. Discovering a path, retaining its evidence and answering correctly are different events. Where the common $7$-billion-parameter proxy is specified, use $F=3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$, weight-stream bandwidth $B=3.35\times10^{12}\,\mathrm{B\,s^{-1}}$ and two bytes per weight. Thus $W_{\mathrm{forward}}\approx2NT$, linear prefill time per token is $2N/F\approx0.041176471\,\mathrm{ms/token}$, and the batch-one weight-stream component is $\tau=2N/B\approx4.179104478\,\mathrm{ms/token}$. These are accounting components, not benchmarks: attention, KV-cache traffic, kernels, utilization, queuing, communication and tool overhead may add cost. The Self-RAG and recursive examples explicitly use different stipulated token rates. Ideal parallel latency requires sufficient concurrent capacity. Prompt positions processed, new input tokens, provider-billed input tokens and output tokens have different denominators. State whether prefixes are replayed, validly cached or billed again. Letters are local to each section: in particular context $C$, chunk size $C$, coverage $C(T)$ and critic $C$ are distinct; so are the FLARE mask threshold $\beta$ and stopping dilution coefficient $\beta_{\mathrm d}$. None of the chapter-reported empirical rates or historical examples is established by the arithmetic alone.

26.1 Why one-shot retrieval breaks

What:

• One-shot retrieval selects context once for a generator call: $C=\mathop{\mathrm{RET}}\nolimits(q,D;k)$, followed by $p(a\mid q,C)$. The selection may itself include ANN search and reranking; “one-shot” does not mean every document is scored exactly once.
• Let $s(q,d)$ be a retrieval score, $f_j$ a required hop fact, $d_j$ its document, $h$ the chain depth and $e$ a bridge entity. A frozen-query system cannot use a newly generated bridge to launch another retrieval unless the control flow provides that return edge. Why the failure appears:
• A chain needs facts $f_1,\ldots,f_h$. Let $E_j$ mean that the selected context contains the evidence for fact $j$.
• The joint evidence probability is a conditional chain product. If $\rho_j$ instead denotes marginal recall, replacing the conditional factors by $\rho_j$ requires independence:

For two independent evidence events with $\rho_1=\rho_2=0.8$, joint coverage is $0.8^2=0.64$. Correlated retrieval errors need not give this value. A missing bridge can lower recall, but absence of a literal term does not prove that dense retrieval, links or a broader first result cannot recover the second document. Failure without it:

• For an explicit information need about sulfur’s classification, suppose penicillin is only a bridge. Passages about the discovery of penicillin can support “Alexander Fleming, in 1928” while failing to answer that need. The error is responsiveness, not necessarily unsupported text; the question must distinguish the two discoveries.
• In “when did he start the company?”, stoplisting may leave “start” and “company.” Their stipulated IDF contributions, $1.10$ and $1.73$, sum to $2.83$. Relative to the self-contained example’s recomputed IDF sum, this is about $15.459149\%$ of the raw IDF total, not a bound on semantic information or answer accuracy. No document frequencies are supplied to rederive those two stipulated contributions.
• A PayPal or Tesla referent already present in turn $t-1$ can be resolved before retrieval; it is not information that inherently arrives after retrieval.
• Topical relevance does not establish current truth. Relevance-only Recall@k can miss staleness; a currency-aware gold set and evaluation can detect it. A high recall score is not, by itself, a certificate of responsiveness or currency. Cost and complexity:
• In a corpus of $10^7$ chunks, use the Robertson–Sparck Jones IDF variant $I(t)=\ln\!\bigl((N-\mathrm{df}_t+0.5)/(\mathrm{df}_t+0.5)\bigr)$, without the additional $1$ inside the logarithm used by another BM25 convention. The document frequencies for penicillin, element, scientist, discovered and found are $4{,}000$, $300{,}000$, $400{,}000$, $800{,}000$ and $2{,}000{,}000$.
• Their IDFs are approximately $7.823520989$, $3.476097075$, $3.178052632$, $2.442346465$ and $1.386294174$. Sum before rounding: $18.306311334$. Penicillin contributes approximately $42.736742\%$ of this raw IDF sum, not a probability or a proven share of discriminative power.
• The other terms sum to $10.482790345$. This is not a BM25 upper bound on a passage lacking penicillin: term-frequency factors can exceed one. For a single occurrence of each query term and $k_1=1.2$, the usual positive-denominator TF factor is bounded above by $k_1+1=2.2$, giving an other-term supremum of about $23.062138760$ in this simplified score. Actual lengths and term frequencies still determine the score.
• With $k=50$ and $400$ tokens per chunk, retrieved text alone has $20{,}000$ tokens, beyond an $8{,}192$-token window. Include a $200$-token question for $20{,}200$ tokens. Linear prefills cost approximately $823.529412\,\mathrm{ms}$ and $831.764706\,\mathrm{ms}$ respectively. A $k=5$ prompt with the question is $2{,}200$ tokens and costs $90.588235\,\mathrm{ms}$.
• For a matched partial schedule, suppose each retrieval costs $83\,\mathrm{ms}$ and a two-hop route generates only a $20$-token bridge before the common final answer. Its cost is $2(83)+2(90.588235)+20\tau\approx430.758560\,\mathrm{ms}$. Wide one-shot costs $83+831.764706\approx914.764706\,\mathrm{ms}$ before the same final answer. The difference is approximately $484.006146\,\mathrm{ms}$, excluding other overhead. The former $224$/$249$ versus $700$ millisecond claims omitted or inconsistently priced these components. This is not the two-full-$80$-token-round scenario in Section $26.2$. Mechanism example:
• In the illustrative chemistry chain, hop one exposes sulfur. At document frequency $30{,}000$, $I(\text{sulfur})=\ln(9{,}970{,}000.5/30{,}000.5)\approx5.806121865$ under the same IDF convention.
• The second query asks who established that sulfur is an element. The reported example points to Lavoisier’s $1777$ argument; this mathematical pass does not independently verify that historical account.
• Distinguish benchmark configurations: the standard ten-paragraph HotpotQA distractor setting contains two supporting paragraphs and eight distractors, not two plus ten. The full-wiki setting is different. A bridge in the first supporting paragraph motivates sequential retrieval; it does not prove that every first query must miss the other paragraph. Practice decisions:
• Resolve available dialogue references before ranking; a second search is not a substitute for reading the history. Evaluate responsiveness, support and currency separately.
• The source’s $300$-example audit and roughly ten-percent multi-hop routing threshold are workflow heuristics, not statistical guarantees. Failure categories can overlap, and a relevance-only grader need not detect stale evidence.
• Raise $k$ only after checking the context and cost budgets. Test whether a bridge-aware rewrite or iterative call improves measured joint evidence retention and downstream answer correctness.

26.2 Iterative retrieval with ITER-RETGEN

What:

• In a toy service graph, one document maps billing-api to prod-ledger-3; another maps prod-ledger-3 to eu-central-1. The bridge $e$ is in both documents but absent from the original query and final region answer.
• Let $r_1$ be first-document recall and $r_{2\mid1}$ second-document recall conditional on obtaining the first. One-shot joint recall is then a chain-rule product, without an independence assumption:

• ITER-RETGEN alternates retrieval and generation:

Why:

• The original question anchors generation while the previous draft augments retrieval. Joining a draft can surface a bridge, but can also amplify a hallucinated bridge; validate the resulting evidence.
• Replacing $D_t$ keeps the passage count at $k$. With query length $Q$, fixed chunk length $C$ and no growing generator history, each generator prompt has $Q+kC$ tokens. This is bounded per round, not prompt size growing linearly with $T$; cumulative full-replay prefill grows linearly. Failure without it:
• If the original query gives no discriminating signal for the second document and ranks are exchangeable, its mean rank among $N=2\times10^6$ chunks is $(N+1)/2=1{,}000{,}000.5$, and uniform top-five inclusion probability is $5/N=2.5\times10^{-6}$. A missing lexical term alone does not justify that exchangeable-rank model.
• A fully specified query naming prod-ledger-3 cannot be constructed from the original wording if that name is genuinely unknown. A placeholder, graph traversal, prior knowledge or a later adaptive step can change the design.
• Static fan-out is useful when subqueries can be prepared before their results arrive, as in many Postgres-versus-MySQL comparisons. A dependent bridge chain may instead need sequential discovery. Evaluate the dependency topology, not merely the label “multi-hop.” Cost and complexity: For full prompt replay, fixed $Q$ and $C$, and $b$ retained generated tokens per prior round, distinguish three prompt-position ledgers. The accumulating external-token ledger excludes those retained outputs; adding them changes the total:

If $D_{h+1}\approx D_h$, stable retrieval is a possible stopping signal, not a proof of a fixed point. Stochastic decoding, changed drafts and retrieval ties can still change later states or create cycles. An extra round still pays for retrieval and generation, and replacement can lose earlier supporting passages. Worked example:

• Fix $N_{\mathrm{chunks}}=2\times10^6$, chunk size $C=400$, question size $Q=200$, $k=5$ and a $7$-billion-parameter fp16 generator. Embedding, ANN search and cross-encoding cost $8+15+60=83\,\mathrm{ms}$ per retrieval in this stipulated ledger; the ANN stage considers $100$ candidates. Hold this retrieval cost fixed despite changed query length only as an explicit simplification.
• Let first-hop recall be $0.86$ and second-hop conditional recall under the original query be $0.09$. Joint one-shot path recall is $0.0774$. Linear prefill of $2{,}200$ tokens costs $90.588235\,\mathrm{ms}$; an $80$-token output costs $334.328358\,\mathrm{ms}$ under the common stream proxy. Total: $507.916593\,\mathrm{ms}$.
• Wide one-shot with $k=50$ uses conditional factors $0.97$ and $0.17$, giving $0.1649$ joint path recall. Its $20{,}200$-token prefill and $80$-token decode total $1{,}249.093064\,\mathrm{ms}$ including retrieval, if the context window can hold it.
• With a correct bridge-extraction probability of $0.90$, the second query contains the correct bridge with probability $0.86(0.90)=0.774$. Conditional retrieval probability $0.78$ then gives discovered-path probability $0.60372$, approximately $3.661128$ times the wide baseline. These are stipulated conditional rates, not measured answer accuracy; ensure the final reader still receives both required facts.
• Two complete $80$-token replacement rounds cost $1{,}015.833187\,\mathrm{ms}$, approximately $233.259877\,\mathrm{ms}$ less than wide one-shot under this ledger. This replaces the inconsistent $134$, $231$ and $992$ millisecond comparisons rather than propagating them as alternative results.
• For seven accumulating rounds, external prompt positions are $57{,}400$. Replaying all prior $80$-token outputs adds $1{,}680$, for $59{,}080$ prompt positions. Linear prefill, decode and retrieval cost approximately $2{,}432.705882$, $2{,}340.298507$ and $581\,\mathrm{ms}$, totaling $5{,}354.004390\,\mathrm{ms}$, or $10.541109$ times the small one-shot ledger. This assumes full replay, no prefix caching and no additional final reader; the former $2{,}009$ millisecond prefill and $4{,}942$ millisecond total do not follow from the stated rate. Practice decisions:
• Begin with a small cap such as two or three rounds, then measure cap hits, path retention and answer correctness. These are tuning defaults, not universal optima.
• Keep earlier source identifiers and re-fetch supporting chunks when a replacement draft is not sufficient evidence. A remembered intermediate answer is not a citation.
• On timeout, return a clearly marked partial result, abstain or hand off as appropriate; do not force a confident answer merely because the cap fired.

26.3 Recursive retrieval with IRCoT and Self-Ask

What: For a deliberately simplified $h$-hop chain, let $p_{\mathrm{on}}$ be conditional retrieval success when the subject is named, $p_{\mathrm{off}}$ the conditional rate when it is not, and $a$ correct bridge extraction conditional on successful evidence. If one-shot is on-subject only for the first hop and these rates stay constant along successful paths:

A recursive loop that correctly names each next subject has the following successful-path probability under the same stationary conditional-rate model, with no retries, alternate paths or repair. It is not yet the final answer probability:

Why:

• Make the example unambiguous: “Who succeeded the U.S. president in office when the Soviet Union dissolved?” The original “who was president after it dissolved?” could mean George H. W. Bush, not necessarily Bill Clinton.
• The reported chain identifies December $1991$ and Bush, then asks who succeeded Bush, leading to Clinton’s January $1993$ inauguration. The mathematical pass does not independently source that chronology.
• The bridge makes a useful next query; it does not prove a knowledgeable retriever or generator could never answer in one step, nor that the November $1992$ election date must be unknown beforehand. Failure without it: With $p_{\mathrm{on}}=0.85$, $p_{\mathrm{off}}=0.25$ and $a=0.90$, two-hop one-shot and recursive path probabilities are $0.2125$ and $0.65025$. At three hops they are $0.053125$ and $0.49744125$; at five hops recursive path success is $0.2911150555$. The powers describe this conditional toy model, not independent verification of an empirical method. IRCoT mechanism:
• In the IRCoT-style implementation, retrieve on $q$, generate one declarative reasoning sentence using $q$, accumulated passages and prior reasoning, and use that sentence as the next query.
• Append the newly retrieved passages. Stop on a chosen terminator such as “so the answer is” or a hard cap, then run a final reader over the collected evidence. The terminator is a model output, not a guarantee of completeness.
• Declarative sentences can resemble corpus prose and suit lexical retrieval, but their effectiveness must be measured. One sentence per step is an implementation choice, not a universal property of every recursive system. Self-Ask mechanism:
• A Self-Ask-style prompt asks whether follow-up questions are needed, generates one single-hop question, retrieves for that question and records an “Intermediate answer.”
• Raw passages can be dropped after extraction while short questions and answers remain in the history; this reduces passage volume but may lose qualifiers, contradictions and provenance. End with a final answer only after checking the retained evidence is sufficient.
• The reported compositionality-gap result concerns the evaluated GPT-3 family and task, not a theorem about every model family or retrieval architecture. Cost and complexity:
• Full-replay IRCoT passage positions grow as $T(T+1)/2$, before adding replayed reasoning and the final reader. Valid prefix caching changes the newly computed positions but not necessarily all attention cost.
• Self-Ask can process a fixed amount of new passage text per hop, hence linear new-document volume. Replaying its growing short question-answer history can still contribute a quadratic term; “shorter” does not mean a constant-size full prompt.
• A catalog pasted into a standard dense-attention prompt is not a B-tree: it generally remains in attention computation rather than pruning disk branches. Sparse attention, learned routing and external indices have different costs, so do not generalize the dense comparison to every architecture. Worked example:
• This separate scenario uses five $200$-token passages, $50\,\mathrm{ms}$ retrieval and $50$ output tokens per second. A one-shot $40$-token answer takes $0.05+40/50=0.85\,\mathrm s$ in retrieval-plus-decode components only; no prefill rate is given.
• Three IRCoT rounds generating $30$ reasoning tokens each, followed by a $40$-token final answer, generate $130$ tokens. Retrieval plus decode is $3(0.05+30/50)+40/50=2.75\,\mathrm s$, or $3.235294$ times the one-shot components. These are not full latency measurements.
• Replayed document positions are $1{,}000(1+2+3)+3{,}000=9{,}000$ including the final reader, excluding question, reasoning and control-token replay. Two Self-Ask follow-ups process $2{,}000$ new document tokens, a $4.5$-fold reduction in this scoped document ledger; a full latency comparison also needs its generated questions, intermediate answers and history processing.
• At $\$0.50$ per million document input positions and one million queries per day, these scoped document bills are $\$500$, $\$4{,}500$ and $\$1{,}000$ per day. They exclude other prompt positions, output billing, tool and serving costs.
• The conditional two-hop model predicts a $43.775$-percentage-point path-coverage gain, from $0.2125$ to $0.65025$. Do not equate it with the reported IRCoT gains of up to $21$ retrieval-recall points or $15$ downstream QA points across HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC.
• Changing $p_{\mathrm{off}}$ to $0.55$ gives one-shot path coverage $0.4675$ and an $18.275$-point gap. The source’s roughly $0.5$ public-benchmark prior is a planning heuristic; measure the actual corpus, queries and conditional rates. Practice decisions:
• Select recursive retrieval when next-query dependencies justify it; select static fan-out when the subqueries are available together. Preserve early support when citations or multi-sentence conditions matter.
• A cap of four retrievals and $k$ around three to five are starting heuristics. Inspect cap-hit failures, compression losses and both retrieval and answer metrics under the actual context and percentile-latency budgets.

26.4 Confidence-triggered retrieval with FLARE

What: FLARE drafts the next sentence $s_t$ of $m$ tokens, using ordinary model probabilities $p_i$ for its drafted tokens under a defined decoding distribution. These are not probabilities that the tokens are true. If no token crosses the threshold, accept the draft; otherwise retrieve and regenerate the affected sentence.

On a trigger, remove or mask uncertain guessed spans when constructing the query, retrieve passages and regenerate $s_t$; discard the fired draft, not every non-triggering draft. Dropping the retrieved passages afterward changes context and may invalidate a cached suffix. Charge that context management where it occurs. Why: A minimum-token gate is sensitive to a single uncertain entity, but rare correct names can have low probability and familiar false statements can have high probability. Longer sentences have more opportunities to trigger. Tune the gate against useful-search and error labels, not just a desired search count. Failure without it: With twenty-four token probabilities of $0.95$ and one of $0.10$, the arithmetic mean is $0.916$. It can hide one weak token; neither the mean nor the minimum is a proof of factuality.

Using natural-log token loss, the same draft has perplexity

For all twenty-five probabilities equal to $0.95$, $H=-\ln0.95\approx0.0512932944$ and $\mathop{\mathrm{PPL}}\nolimits=1/0.95\approx1.052631579$. The relative perplexity gap is about $9.423082\%$. This demonstrates averaging dilution, not a calibrated factuality threshold; rounded exponents must not be connected by exact equality. Implicit and explicit queries:

• An implicit query masks tokens below the mask threshold $\beta$, for example a draft of “Person X attended [mask] and earned [mask].” It adds no extra query-generation call beyond the already charged draft. Retrieval still costs work, and the remaining frame may be uninformative or combine distinct needs.
• An explicit query generates one question per uncertain span, not necessarily one per token. It separates needs but adds query decoding and input processing.
• The trigger threshold $\theta$ and mask threshold $\beta$ have different roles. With the same strict comparison, $\beta\ge\theta$ covers every triggering token; if $\beta\lt \theta$, a trigger may mask nothing. Specify span expansion and threshold-boundary handling.
• This implementation needs access to the relevant token probabilities. An API without them needs a different available gate or model interface; a trained Retrieve control token is an alternative architecture, not a feature automatically present in the existing model. Cost and complexity:
• Hold answer length at twelve sentences of twenty-five tokens, with the common stream proxy $\tau\approx4.179104478\,\mathrm{ms/token}$. Sentence decode is $S=25\tau\approx104.477612\,\mathrm{ms}$ and the $300$-token output costs $1{,}253.731343\,\mathrm{ms}$.
• A retrieval path has $8\,\mathrm{ms}$ embedding, $15\,\mathrm{ms}$ ANN and $500$ new retrieved tokens. Their linear prefill costs $20.588235\,\mathrm{ms}$, so $R\approx43.588235\,\mathrm{ms}$.
• Retrieve-once and retrieve-before-every-sentence components total approximately $1{,}297.319579$ and $1{,}776.790167\,\mathrm{ms}$. The latter makes twelve searches and injects $6{,}000$ raw retrieved tokens.
• Implicit FLARE with four triggers reuses the other drafts and adds four regenerated sentences: $12S+4(S+R)\approx1{,}845.994732\,\mathrm{ms}$. It makes four searches and injects $2{,}000$ raw retrieved tokens.
• Six explicit twelve-token query generations add $72\tau\approx300.895522\,\mathrm{ms}$, for $2{,}146.890255\,\mathrm{ms}$, before their extra input processing. These partial ledgers omit shared answer-history processing, cache changes and other serving overhead; they are not complete wall-clock latencies. Break-even criterion:

• The break-even assumes every non-triggering draft is reused, each triggered draft adds one equal-length regeneration, and all other costs match. An explicit-query cost $d$ per trigger changes it to $f^\star=R/(S+R+d)$.
• Here $f^\star\approx0.294384128$. Four triggers in twelve sentences are above it, so implicit FLARE is approximately $69.204565\,\mathrm{ms}$ slower than every-sentence retrieval in these components, while using one third of the searches and raw injected tokens.
• If the entire retrieval path is instead $R=400\,\mathrm{ms}$, $f^\star\approx0.792899408$. If $400\,\mathrm{ms}$ is just web wait, add embedding and prefill before applying the formula.
• The bandwidth calculation is a weight-stream roofline, not measured token latency. Achieving $70\%$ to $85\%$ of that bandwidth would give approximately $5.970149$ to $4.916593\,\mathrm{ms/token}$ for this component. This does not establish an actual batch-one performance range.
• With $\tau=6\,\mathrm{ms}$ and $R=40\,\mathrm{ms}$, break-even is $0.210526316$. With the original stream proxy and $R=40\,\mathrm{ms}$ it is $0.276859504$. Different rate assumptions must not be silently combined into one threshold. Practice decisions:
• Tune trigger and mask policies on sentence lengths, calibration and false positives; low token confidence alone is not missing-evidence detection. Mask uncertain guesses without unnecessarily erasing grounded user entities.
• A twelve-token explicit query costs approximately $50.149254\,\mathrm{ms}$ in this decode component, before its prefill and tool work. Charge all queries generated for multiple spans.
• A cap such as four searches is a product heuristic. Benchmark whole-answer p95, context replay and cache invalidation rather than inferring them from summed mean components.

26.5 Corrective RAG

What:

• CRAG places a separately trained retrieval evaluator between retrieval and generation. It aggregates query-document scores into a set score $s$; this is not a calibrated correctness probability without validation.
• Choose $\tau_{\mathrm{lo}}\lt \tau_{\mathrm{hi}}$. Here define INCORRECT as $s\lt \tau_{\mathrm{lo}}$, AMBIGUOUS as $\tau_{\mathrm{lo}}\le s\le\tau_{\mathrm{hi}}$, and CORRECT as $s\gt \tau_{\mathrm{hi}}$, so every boundary has a branch.
• CORRECT splits chunks into short strips, rescores and recomposes survivors. INCORRECT discards local results, rewrites the query and filters web results. AMBIGUOUS retains filtered local evidence while adding filtered web results.
• Strip filtering can remove a needed qualifier, and appended evidence can distract. These are architectural mitigations, not correctness guarantees. The reported Self-CRAG system combines this correction layer with Self-RAG. Why: For a binary gate, let $p$ be useful-local-evidence presence, $r$ detection probability given a local miss, $f$ false-discard probability given a useful local hit, and $\alpha,\alpha_0$ useful-web-evidence presence after those two types of escalation. Assume constant answer accuracy $g$ when useful evidence is present and zero otherwise. Then the answer-accuracy change is

• The bracket alone is a change in useful-evidence presence under this model, not answer accuracy unless $g=1$. The first term is rescued misses and the second useful hits lost after false discard and failed web recovery.
• Sensitivity to $f$ relative to sensitivity to missed detection $1-r$ is $p(1-\alpha_0)/[(1-p)\alpha]$. At $p=0.72$, $\alpha=0.60$ and $\alpha_0=0.50$, it is $2.142857$, not $2.5$; it depends on the web rates, not $p$ alone.
• An ambiguous branch that preserves all needed local support can reduce discard harm. Assuming its answer accuracy stays unchanged is an additional reader assumption: longer or conflicting context can still cause harm. Failure without it:
• Retriever similarity measures a ranking objective, not a certified answer-presence probability. It can correlate with containment, but an unanswerable query still has a nearest neighbor.
• A generator grading its own output can share its original blind spots. A separate evaluator changes the model and training objective, but does not establish independent errors or guaranteed decorrelation.
• The chapter reports a separately fine-tuned T5-large evaluator of approximately $770\times10^6$ parameters fitted on PopQA. Calibrate thresholds and check transfer on the new domain; reported scores are not automatically probabilities there. Cost and worked example:
• Stipulate $p=0.72$, answer accuracy $g=0.85$ when useful evidence is present and zero otherwise, evaluator $r=0.85$, $f=0.08$, and useful-web-presence rates $\alpha=0.60$, $\alpha_0=0.50$. A web call costs $\$5$ per thousand calls and $400\,\mathrm{ms}$ of wait.
• Baseline accuracy is $pg=0.612$. Escalation probability is $(1-p)r+pf=0.2956$, not $0.296$ before final rounding. Binary-gate accuracy masses are $0.56304$ retained local, $0.12138$ rescued local misses and $0.02448$ recovery after false discard, totaling $0.7089$.
• Move three quarters of evaluator false positives to AMBIGUOUS and assume their preserved support remains usable with the same $g$. This contributes $0.0432(0.85)=0.03672$; the remaining false-discard branch contributes $0.0144(0.50)(0.85)=0.00612$. Together with retained hits and rescued misses, accuracy is $0.72726$, a gain of $11.526$ points over baseline and $1.836$ points over the binary gate.
• Five evaluator pairs of $230$ tokens process $1{,}150$ positions. The forward proxy is $1{,}150(2)(0.77\times10^9)=1.771\times10^{12}\,\mathrm{FLOP}$, or $5.208824\,\mathrm{ms}$ at $F$. This is $6.325\%$ of the $82.352941\,\mathrm{ms}$ linear prefill of $2{,}000$ tokens through the $7$-billion-parameter generator, not a full serving-overhead ratio.
• At $\$2.50$ per GPU-hour, a thousand evaluator runs cost approximately $\$0.003617$. Web cost is $0.2956(\$5)=\$1.478$ per thousand user queries, for $\$1.481617$ total incremental cost. Divide by $115.26$ net additional correct answers to obtain approximately $\$0.012855$ per net improvement; that denominator is not a count of gross wrong-to-right conversions.
• Mean incremental components total $5.208824+0.2956(400)\approx123.448824\,\mathrm{ms}$. If baseline latency is a common constant and the web delay is exactly two-point, the added p95 is $405.208824\,\mathrm{ms}$. General p95s cannot be obtained by adding component quantiles; web latency variation and dependence require measurement.
• The reported Self-CRAG PopQA score of $61.8\%$, roughly seven points above its own Self-RAG baseline, is a separate experimental result, not validation of this constructed $72.726\%$ scenario. Practice decisions:
• Fit and validate thresholds on local labels; $300$ to $500$ queries are the source’s starting heuristic, not a coverage guarantee. Report both false discard of useful sets and failure to escalate unhelpful ones.
• Filtering chunks above roughly $200$ tokens is another heuristic. Keep tables, code, negations and cross-sentence qualifiers intact when strip boundaries would destroy their meaning.
• Under overload, local-only serving is not inherently safe. Depending on evidence, access policy and product obligations, abstention, partial answers or handoff may be preferable.

26.6 Self-RAG

What: Self-RAG extends the generator vocabulary with learned reflection groups: Retrieve decisions, IsRel relevance, IsSup support and IsUse ordinal usefulness ratings. For a specified desired token $r_G^\star$ in group $G$, normalize its probability within that group, with a positive denominator:

For $K$ passage-conditioned candidate segments, a schematic decode-time ranking combines segment likelihood with weighted reflection scores. Specify the target reflection token or rating expectation used in each group; IsUse can be an expected ordinal score rather than an argmax token. One illustrative score is

The weights $w_G$ are decode-time choices. Group-normalized model scores are not automatically calibrated truth probabilities. Use a consistent sequence-likelihood or length-normalized convention across candidates; otherwise unequal segment lengths can distort the ranking. This schematic is not a claim that every Self-RAG implementation uses exactly this score. Why: Reflection tokens allow structured decisions without a second free-form critique prompt or string parser, and can place retrieval/support checks within generation. They still require decoding, supervision and validation. FLARE’s ordinary token-confidence gate and a learned support-token score are different signals, neither a factuality guarantee. Training and claim limit:

• In the reported training recipe, an expensive teacher labels a seed set, a critic $C$ is distilled from it, $C$ annotates the generator corpus offline and generator $M$ learns content plus reflection tokens with next-token training. The offline critic is not called at serving time.
• A scenario with $20{,}000$ teacher calls and $150{,}000$ local critic calls uses $7.5$ times fewer teacher calls than directly labeling all $150{,}000$ examples with the teacher. This does not mean a $7.5$-fold reduction in total training cost.
• Noisy teacher judgments can limit the student, but there is no universal theorem that a distilled student cannot outperform its teacher on a task after other data or learning. Content and critique sharing serving weights can correlate errors; separate CRAG weights do not establish statistical independence either. Cost and worked example:
• This separate scenario stipulates $50$ output tokens per second per active stream and $8{,}000$ prefill positions per second. Use $K=5$, $200$-token passages, a $100$-token question/instruction and four $30$-token answer segments. Vanilla RAG processes $1{,}100$ prompt positions and $120$ output tokens: $1{,}100/8{,}000+120/50=2.5375\,\mathrm s$.
• Charge $30$ content plus three reflection tokens per candidate segment. All five candidates per segment give $4(5)(33)=660$ decoded tokens. This counts the specified three reflections only; any extra closing or control tokens must be added in an implementation.
• A declared cache/reseed schedule processes $5(100+200)=1{,}500$ initial prompt positions and $3(5)(33)=495$ winner-prefix refresh positions, for $1{,}995$ positions and $0.249375\,\mathrm s$ prefill. It keeps one winner per segment, not an exponentially growing beam. Charging every branch refresh is a conservative choice; the winning branch may reuse its valid KV state, while replaying without caches costs more.
• Only if five-way batching maintains $50$ tokens per second per stream does decode take $4(33)/50=2.64\,\mathrm s$. Total is then $2.889375\,\mathrm s$, or $1.138670$ times vanilla latency. Without this extra capacity, serial decode is $660/50=13.2\,\mathrm s$, totaling $13.449375\,\mathrm s$ under the same prefill ledger. A $13.867\%$ latency premium is conditional, not a measured universal result.
• Decode work is $660/120=5.5$ times baseline. Of the $540$ extra tokens, $60$ are reflection and $480$ discarded candidate content; reflection is $11.111111\%$ of extra decode. At $K=2$, decode is $264$ tokens, or $2.2$ times baseline work, not necessarily latency.
• As a component-only comparison, extra decode uses $7(540)=3{,}780$ billion-parameter-token units. A $0.77$-billion-parameter evaluator processing $1{,}500$ input positions uses $1{,}155$ such units, a ratio of $3.272727$. This excludes Self-RAG’s extra $895$ prefill positions, attention, batching differences and control costs; it is not a full cost comparison. The earlier CRAG example used $1{,}150$, not $1{,}500$, evaluator input positions. Practice decisions: Choose $K$ and reflection weights on held-out quality, support, latency and cost measurements. A support label is a learned judgment, not proof. Tune the branch budget to available capacity, preserve citations for the selected branch and report total candidate work alongside wall time.

26.7 Search-R1, GRPO, and learned stopping

What:

• In the described Search-R1 interface, the policy generates reasoning, search requests and final answers. A search closing tag pauses the policy, E5 retrieval returns the top three Wikipedia passages inside information tags and decoding resumes.
• Retrieved information is an environment observation conditioned on by later policy actions; the policy did not author the search engine’s text. Distinguish the interleaved transcript from the action sequence.
• The source describes a baseline with exact-match terminal reward and no explicit format, process or search-count reward. That is a version-specific recipe, not a restriction on every search-trained policy. Treat retrieved text as untrusted tool data, not new instructions. Why GRPO:
• PPO commonly uses a learned value baseline; GRPO instead compares $G$ sampled rollouts for one question. Sampling groups has a cost even when the baseline is a simple mean.
• For an unsharded $7$-billion-parameter bf16 critic under the specified AdamW accounting, weights and gradients are $14\,\mathrm{GB}$ each, while fp32 master weights and two fp32 moments are $28\,\mathrm{GB}$ each. Total training state is $112\,\mathrm{GB}$, excluding activations, caches, actor and reference models.
• This is not unavoidable PPO critic memory: critic size, shared trunks, sharding, optimizer precision and offload change it. Define the population-standard-deviation group advantage as follows, setting it to zero for an exactly constant-reward group:

• For binary rewards and $c$ successes, let $\hat p=c/G$ be the empirical group success fraction, distinct from a true sampling probability. If $0\lt c\lt G$, success advantage is $\sqrt{(1-\hat p)/\hat p}$ and failure advantage is $-\sqrt{\hat p/(1-\hat p)}$ under the population variance above.
• At $G=5$ with one success, advantages are $+2$ and four copies of $-0.5$. With two successes, the positive value is $\sqrt{1.5}\approx1.224745$ and the negative value is $-\sqrt{2/3}\approx-0.816497$, with $\sigma=\sqrt{0.24}\approx0.489898$.
• Advantages sum to zero in a nonconstant group, but the weighted policy gradients do not generally cancel. Using sample variance, an epsilon or a different normalization changes the magnitudes. Group-relative normalization is not automatically an unbiased gradient estimator for expected exact match. Failure without safeguards: If every reward is identical, these reward advantages are zero. This eliminates the reward-advantage term, not necessarily the entire training gradient: KL regularization, entropy, auxiliary losses, weight decay or optimizer state can still cause updates. For conditionally IID binary-reward rollouts with true success probability $\pi$, the probability of a constant-reward group is

• At $G=5$, constant-reward probabilities are $0.0625$ for $\pi=0.5$, $0.5905$ for $\pi=0.9$ and $0.77378125$ for $\pi=0.95$. At $\pi=0.9$, $G=16$ gives approximately $0.185302019$. Correlated samples or nonbinary total rewards need a different calculation.
• An $810$-token transcript with $40$ prompt, $170$ policy-generated and $600$ retrieved tokens has a retrieved fraction $600/810\approx74.074074\%$.
• Mask prompt and retrieved observation positions out of the policy-action loss and the action-level policy/reference KL penalty. Keep them in the model context so later policy actions can depend on them; masking loss does not erase context or its forward cost.
• Scoring the search engine’s observed tokens as policy actions would add an imitation/language-model objective, not the standard KL over controllable actions. Observation text being legitimate context is not a reason to penalize its token likelihood as if the policy chose it. Search cost objective:
• Exact-match reward alone does not price search. An alternative is $r=\mathrm{EM}-\lambda n$ for $n$ searches, with reward and $\lambda$ in consistent utility units.
• Let $v_t$ be expected terminal correctness under a specified continuation policy. In a myopic comparison of exactly one extra search, holding other costs and continuations fixed, search is worthwhile if $v_{t+1}-v_t\gt \lambda$. This is not a theorem that RL finds the optimal stopping policy; a full Bellman comparison also accounts for future search and costs.
• For a $3$-billion-parameter model, a $300$-token linear prefill costs $300(6\times10^9)/F\approx5.294118\,\mathrm{ms}$. Add $20\,\mathrm{ms}$ retrieval for $25.294118\,\mathrm{ms}$, excluding search-query decode and other overhead.
• If one accuracy percentage point is worth $100\,\mathrm{ms}$, one unit of accuracy fraction is worth $10{,}000\,\mathrm{ms}$. The corresponding component-only penalty is $\lambda\approx25.294118/10{,}000=0.002529412$, not a conversion using milliseconds per unit fraction as if it were per point. Training cost:
• Use a deliberately coarse arithmetic training ledger for a $3$-billion-parameter policy: $2N=6\times10^9$ forward FLOPs per position and $6N=1.8\times10^{10}$ forward-plus-backward FLOPs per position. The $810$-position trace has $170$ generated and $640$ prompt/observation positions. Rollout decode costs $1.02\times10^{12}$ FLOPs; rollout prefill $3.84\times10^{12}$; reference forward over the trace $4.86\times10^{12}$; policy forward/backward $1.458\times10^{13}$. Total is $2.43\times10^{13}\,\mathrm{FLOP}$ per rollout. Loss masks do not remove conditioning-context computation.
• At $F$, this is $71.470588\,\mathrm{ms}$ of arithmetic time per rollout, not sequential batch-one wall time. Five rollouts cost $357.352941\,\mathrm{ms}$ per question in this ledger. A $100{,}000$-question pass costs $9.926471$ arithmetic GPU-hours, or $\$24.816176$ at $\$2.50$ per GPU-hour; tool waits, attention, communication and utilization are omitted.
• With conditionally IID groups, a prompt mixture of $30\%$ at $\pi=0.95$, $25\%$ at $\pi=0.05$ and $45\%$ at $\pi=0.5$ gives $45.370469\%$ constant-reward groups. Expect $54{,}629.53125$ nonconstant groups, or $5{,}503.419444$ per arithmetic GPU-hour. Approximately $4.503686$ hours of the full ledger belongs to constant-reward groups; it is not necessarily wasted regularization signal.
• Generation/prefill costs $14.294118\,\mathrm{ms}$ per rollout and reference/update $57.176471\,\mathrm{ms}$. Only if the latter stages have no required nonzero losses and can be skipped without changing normalization, optimizer-state behavior or the target objective does the optimistic ledger save $36.296375\%$, reaching $6.323522$ hours and $\$15.808804$. Equal reward alone does not justify “identical gradients”; an active KL term is a direct counterexample.
• An oracle that discards the entire $55\%$ extreme-probability prompt cohort retains $45{,}000$ balanced prompts. Its training ledger costs $4.466912$ hours and yields $42{,}187.5$ expected nonconstant groups, or $9{,}444.444444$ per hour, $1.716105$ times the original rate. It loses $22.775285\%$ of the original nonconstant groups and changes the prompt distribution.
• A real five-rollout probe does not identify that oracle cohort: it retains approximately $54.629531\%$ after filtering observed constant-reward groups, not $45\%$. Generating probes for all original prompts alone costs approximately $1.985294$ arithmetic hours in this ledger unless validly reused. Reusing rollouts, drawing fresh groups and resampling prompts are different algorithms with different objectives and costs; probe compute is not free.
• A weight-only arithmetic/bandwidth balance gives batch size $F/B\approx101.492537$, hence at least $102$ in that toy comparison. It is not a guarantee that batch $128$ is optimal or fits memory; attention, KV traffic and variable lengths matter. Increasing $G$ from five to sixteen multiplies linear rollout work by $3.2$, while reducing constant-group probability at $\pi=0.9$ as above. Measure useful learning per resource, not group count alone. Practice decisions:
• Log reward variance, constant-group rate, search count, KL, tool errors and held-out answer performance. Ten-percent or forty-percent constant-group thresholds are source heuristics dependent on group size, correlation and reward shaping, not universal signals to change an optimizer.
• Dynamic sampling changes the training distribution and can remove informative examples. Charge its probes and establish which loss terms remain on filtered groups before claiming gradient equivalence or savings.
• Keep tool permissions, timeouts and hard search caps. A cap derived from twice an observed p99 trace length is a resource heuristic, not a safety guarantee or proof that the learned stopper is optimal.

26.8 Agentic RAG and its costs

What and why:

• A sub-agent has its own context and a configured retrieval interface or permission scope; a controller dispatches work and combines summaries or structured evidence. Interfaces need not be exclusive unless that is an explicit policy.
• Let $\ell$ be layer count, $d$ hidden width and $L$ context length. A full-square attention prefill proxy is $W(L)=2NL+4\ell dL^2$. For a fixed total of $L$ tokens split equally across $m$ contexts with no duplicated overhead, $mW(L/m)=2NL+4\ell dL^2/m$. Only the quadratic term shrinks in that comparison. Causal kernels can change its coefficient.
• With $N=7\times10^9$, $\ell=32$ and $d=4{,}096$, the coefficient is $4\ell d=524{,}288$. A $16{,}300$-token monolith costs $2.282\times10^{14}$ linear plus $1.3929807872\times10^{14}$ attention FLOPs, totaling $3.6749807872\times10^{14}$, or $1{,}080.876702\,\mathrm{ms}$ at $F$.
• Four agents each process $4{,}000$ retrieved tokens plus a duplicated $300$-token prompt. Each prefill costs $6.989408512\times10^{13}$ FLOPs, or $205.570839\,\mathrm{ms}$. That is the ideal concurrent prefill time only with sufficient resources; on one shared constrained device, four requests need not finish in one request’s time.
• This attention-aware example differs from the earlier linear-only prefill ledgers. Add duplicated prompts, summaries, controller work and tool waits before comparing complete schedules. Failure and crossover:
• Each $200$-token summary costs approximately $835.820896\,\mathrm{ms}$ under the stream proxy, and controller prefill of $1{,}100$ tokens costs $47.159966\,\mathrm{ms}$ under the attention proxy. Their $882.980862\,\mathrm{ms}$ is an added component, not the net fan-out tax.
• Splitting initial prefill already saves $1{,}080.876702-205.570839=875.305864\,\mathrm{ms}$ on the ideal critical path. Net non-loop overhead here is therefore only $\Delta_0\approx7.674998\,\mathrm{ms}$, before other scheduling differences.
• Let $W_s^{\mathrm{mono}}$ and $W_s^{\mathrm{agent}}$ denote additional adaptive follow-up durations for source $s$ under their respective context schedules. If those follow-ups serialize in the monolith but overlap across agents, the conditional crossover is

• In the further simplification of equal per-round cost $c$ and $n_s$ additional follow-ups per source, this becomes $c(\sum_s n_s-\max_s n_s)\gt \Delta_0$. Initial retrieval waits that can already run concurrently in either architecture must not be counted as saved serial follow-ups.
• The former “more than $1.08$ rounds” threshold follows only from separately ignoring the initial prefill advantage and taking $\Delta_0=882.980862\,\mathrm{ms}$, $c=815\,\mathrm{ms}$: the quotient is $1.083412$. It is not the crossover for the worked context-splitting scenario or a universal requirement for agents.
• Fixed concurrent retrieval can be implemented without agents. Agent autonomy is useful when per-source adaptive queries, state or permissions justify it; context partitioning can also help even without follow-ups. Compare the full feasible baseline, not a deliberately serial fixed-retrieval implementation. Worked example:
• Use four sources, the attention-aware $7$-billion-parameter proxy, $4{,}000$ retrieved tokens per source, $300$ prompt tokens per initial call, $200$-token agent summaries and a $150$-token final answer. Input/output price proxies are $\$3$/$\$15$ per million tokens. Initial source waits of $400\,\mathrm{ms}$ run concurrently in both architectures and are charged once on each critical path.
• With no follow-ups, the monolith’s prefill plus answer is $1{,}707.742374\,\mathrm{ms}$, or $2{,}107.742374\,\mathrm{ms}$ including initial wait. Four ideal concurrent agents plus controller take $1{,}715.417372\,\mathrm{ms}$ in compute/decode components, or $2{,}115.417372\,\mathrm{ms}$ including wait. Agents are $7.674998\,\mathrm{ms}$ slower here. Device-busy components, excluding reservation during waits, are $4{,}839.592574$ versus $1{,}707.742374$ device-milliseconds, a $2.833913$ ratio.
• Non-loop token-price proxies are $\$0.05115$ for $16{,}300$ input and $150$ output tokens, versus $\$0.06915$ for $18{,}300$ input and $950$ output tokens across agents and controller. The premium is $35.190616\%$; ideal parallel latency does not make aggregate work free.
• Now add one $30$-token adaptive follow-up query at each of two sources. For new evidence block size $b$ after prefix length $L$, declare an append-only prefill proxy $W_{\mathrm{append}}(L,b)=2Nb+(4\ell d)b(L+b)$: compute new queries against old/new keys rather than replaying old queries. This block proxy, the full-square initial proxy and weight-only decode remain approximations; decode attention is omitted.
• The serialized monolith follow-ups cost approximately $815.476370$ and $840.333789\,\mathrm{ms}$, including query decode, $400\,\mathrm{ms}$ wait and a $4{,}000$-token append after prefixes of $16{,}330$ and $20{,}360$ tokens. Each looping agent’s corresponding work is $741.459241\,\mathrm{ms}$ after a $4{,}330$-token prefix, and the two agent follow-ups overlap under the capacity assumption.
• Totals including the common initial wait are $3{,}763.552533\,\mathrm{ms}$ for the monolith and $2{,}856.876613\,\mathrm{ms}$ for agents: $24.090960\%$ faster, not $27\%$ of the full schedule. Excluding that common initial wait gives partial components of $3{,}363.552533$ and $2{,}456.876613\,\mathrm{ms}$; label that narrower denominator if using it.
• Under a new-input-only price proxy with reused prefixes billed free, the looping monolith has $24{,}300$ new input and $210$ output tokens, costing $\$0.07605$. Agents have $26{,}300$ and $1{,}010$, costing $\$0.09405$, a $23.668639\%$ premium. Free billing of reused prefixes is an assumption, not standard API pricing.
• Under full repeated-prompt input billing with no cache discount, the same logical calls bill monolith inputs of $16{,}300+20{,}330+24{,}360=60{,}990$, costing $\$0.18612$ including outputs. Agents bill $2(4{,}300+8{,}330)+2(4{,}300)+1{,}100=34{,}960$ input tokens, costing $\$0.12003$ including outputs: $35.509349\%$ cheaper. Billing assumptions reverse the ordering; computational prefix reuse and provider billing are distinct.
• A reported research system’s approximately fifteen-times-chat versus four-times-chat token counts give a ratio $15/4=3.75$ only with a common denominator. They do not identify agent fan-out or validate this dollar ledger. Do not multiply an already-looped dollar cost by another round count to manufacture agreement with a token ratio.
• Four required agent successes at independent probability $0.9$ give $0.9^4=0.6561$. Correlated or conditional success needs a joint model. A suggested metrics layout with two scoreboards per agent plus dispatch and synthesis has $2m+2=10$ scoreboards for four agents; it is a design choice, not an attribution minimum. Practice decisions:
• Start with shallow dispatch and bounded summaries if they fit the task; source suggestions such as depth one and two-hundred-token summaries are heuristics, not guarantees of fidelity or optimum cost.
• Benchmark against concurrent fixed retrieval and a cache-aware single controller. Record aggregate device work, token billing, latency percentiles, tool permissions, missing citations and summary losses separately.
• Put timeouts and structured evidence requirements on each agent. A partial or failed source result must remain visible in the final answer; doubling a measured p99 timeout is not proof that all required evidence will arrive.