Chapter 35: Source Credibility

00-START-HERE · 01-READING-ROUTES · 34_evaluating_the_system · 36_provenance_and_adversarial_robustness Purpose: separate source agreement, document credibility, claim freshness, intent coverage and group exposure; explain what each intervention actually consumes and guarantees; and check the mathematics before comparing methods or making deployment promises.

TL;DR

• Trustworthiness is multidimensional and cross-stage. A stage-oriented checklist routes investigation; it does not prove a unique component caused a failure.
• An on-topic document can be unreliable, and misinformation can mislead generation. Attention is not uniform by default, and no universal twenty-to-thirty-point loss follows from adding one weak document.
• RA-RAG estimates agreement with a latent consensus on non-abstaining source answers, then uses transformed vote weights and source-specific inference. Agreement is not independently verified truth or immunity to copied/Sybil sources.
• CAG fine-tunes generation on credibility-annotated inputs and teacher explanations. It still consumes credibility annotations at inference; it is not a content-only truth scorer.
• CrAM uses known misinformation examples to select influential heads and renormalizes credibility-weighted attention in those heads. It needs upstream scores and model-internal access, not just a text-generation API.
• Treat resource gates as feasibility checks, not a universal method ranking or a same-day deployment guarantee. None of these methods alone resolves freshness, intent coverage or exposure policy.
• Use correctly dimensioned soft-ranking constraints, test diversity optima rather than asserting them, and state that sampled rankings satisfy exposure constraints in expectation—not on every displayed list.

The story

Picture a newsroom preparing a public briefing from a crowded clipping archive. The assignment desk is the retriever. The reporter is the generator. The fact-check desk is the checking stage. The newsroom tracks six trust questions across all three desks. Retrieval is a useful checkpoint for robustness and privacy, generation for fairness and transparency, and checking for factuality and accountability. These are starting points, not exclusive ownership: privacy also depends on generation and access control, and noisy-input robustness depends on the reporter’s reading. A score does not identify one culprit without investigation. The assignment desk also learns that relevance is not credibility. A polished outdated clipping can mislead the reporter even among sound material. The effect depends on the question, placement, conflicts, reader and scoring rubric. Fewer clippings may help or harm; a relevance cutoff is not a validated credibility detector. The newsroom has three ways to use a credibility signal. RA-RAG keeps source identities, checks source-specific answers against a learned consensus and weights their votes. CAG trains the reporter to use externally annotated credibility while explaining an answer. CrAM intervenes in selected attention channels whose credibility-based modification reduced the probability of known wrong answers on calibration examples. The editor checks prerequisites before choosing a method. RA-RAG needs meaningful source identities and calibration coverage; CAG needs usable annotations, training resources and credibility-annotated inference inputs; CrAM needs calibration examples, an upstream scorer and access to attention internals. If those inputs are missing, the deadline does not make any method feasible. Even a well-run newsroom can fail. A statistic measured in 2020 and repeated by a 2023 summary can appear recently measured if publication and observation dates are confused. Follow provenance to the measurement or substantive re-verification, not merely the earliest page or the newest wrapper. Five excellent clippings about the basketball player can miss another Michael Jordan. On an ambiguous query, coverage of plausible entities can be valuable. Once the user explicitly requests the researcher, irrelevant names are not beneficial diversity; refine relevance or clarify the intent. Large wire services can occupy every visible slot while other eligible sources receive none. A rotation policy can allocate expected rank attention under a chosen fairness rule. Each user still sees one concrete list; finite samples can deviate from the target, so actual exposure must be monitored. The newsroom lesson is to separate the questions. Trust, credibility, freshness, coverage, and exposure interact, but no one score or method answers all five.

Decoder table

Core mechanics

Mathematical conventions

Let a credibility score be a declared measurement or model output, not an automatic probability of truth. Specify the unit (source, document, sentence or claim), domain, observation date, abstention policy, label provenance and validation set. Relevance, source reputation, timeliness and factual accuracy are different quantities even when a method combines them. Historical experimental scores below name their model, benchmark, metric and setting. They are reported results, not replicated measurements or a cross-paper ranking. Accuracy/EM differences are percentage points; relative improvements use the baseline as denominator. Accuracy marginals identify a net change in correct answers, not the number of individual answer flips. For attention, weights refer to one specified head and query-token row after softmax; distinguish key-token multipliers, document-conditional shares and full-row shares. Use nonnegative multipliers, preserve causal masks and define fallbacks for zero denominators. Access to attention internals is a separate prerequisite from “no fine-tuning.” For diversity, declare the candidate universe, nonzero embeddings, intent set, cardinality budget and objective scaling. For exposure, distinguish displayed positions from documents, average per-document group exposure from group-total exposure, and expected constraints from finite-run guarantees. All toy utilities, discounts, rates and timing assumptions are stipulated; keep full precision until reporting.

35.1 Six aspects across three stages: a trustworthiness taxonomy

Stage-aware trust

• What: Track robustness, privacy, fairness, transparency, factuality and accountability across retrieval, generation and checking.
• Why: Stage tags support investigation and incident ownership, not a one-to-one mathematical attribution of cause.
• Limit: A $2\%$ sentence-level hallucination rate alone does not measure unauthorized retrieval, identity leakage, viewpoint treatment, provenance or redress; its judge and denominator matter too.
• Cost: Maintain separately defined evaluations and failure gates rather than averaging incomparable raw scores. Laundered provenance can harm transparency and factuality. A missing relevant citation can harm coverage and accountability. Neither observation locates a unique failing stage, and absence from a reference list does not by itself prove an author saw and suppressed that source.

Survey-style evaluation and radar area

• What: Zhou et al.’s 2024 survey organizes RAG trustworthiness into six dimensions and runs scoped prompt-based evaluations.
• Why: A profile can expose a weak dimension hidden by an aggregate.
• Limit: Its dimensions use different tasks, scales and directions; some models have unmeasurable entries because they do not follow the task. They are not automatically six comparable $[0,1]$ scores.
• Cost: The transparency protocol checks generated intermediate key facts against references. Such overlap measures explanation content, not certified faithfulness of a model’s internal computation.[1] The original survey evaluates more than three models. Its Table II reports, for example, GPT-3.5-Turbo privacy refusal $0\%$ and GPT-4o $4\%$ on the constructed Enron-email task, while their factuality scores are $40.0$ and $26.0$ under that task’s metric. GPT-4o is not best on every axis. Llama2-13B has missing robustness/fairness entries, not measured uniformly low scores. These are historical protocols, not general present-day privacy or model-quality guarantees.[1] For a separate illustrative radar chart, assume six nonnegative, commensurately scaled scores $r_1,\ldots,r_6\in[0,1]$ on equally spaced axes in a fixed order. Close the polygon with $r_7=r_1$; using $r_{(i+1)\bmod6}$ with one-based labels would incorrectly introduce $r_0$. Then:

Toy Model A has scores $(0.80,0.30,0.75,0.70,0.85,0.60)$. Its adjacent products sum to $2.575$, so its area is approximately $1.115007707$. Model B has $0.65$ on every axis; its products sum to $2.535$ and its area is approximately $1.097687199$. Relative to B, the difference is $2.575/2.535-1\approx1.577909\%$, below $2\%$. Normalized areas are $0.429166667$ and $0.4225$. Similar areas conceal different component risks. Area also changes with axis order and scaling; it is not a probability or a trustworthy-model certificate.

Operating rules

• What: Report defined component metrics, missing entries, uncertainty and stage tags; keep privacy and other hard requirements independently visible.
• Why: A low-risk average cannot compensate for a disqualifying exposure or safety failure.
• Limit: Restricting retrieved data does not prove sensitive information can never enter prompts, weights, outputs, tools or logs.
• Cost: A five-axis deployment plot is acceptable only as a clearly labeled view with separately audited privacy controls; its area is not directly comparable with a six-axis plot. Re-evaluate affected dimensions when generators, retrievers, access policies, scorers or corpora change. Route an incident using measured evidence and intervention tests rather than assigning it solely from an axis label.

35.2 One bad document among many good ones

Relevance is not credibility

• What: A relevance-oriented baseline selects documents using $\mathrm{rel}(q,d)$ and generates $a=\mathrm{LLM}(q,d_1,\ldots,d_k)$.
• Why: Query match and evidence reliability answer different questions.
• Limit: A dot product, cosine threshold or relevance reranker does not by itself certify credibility, even if it correlates with it on a particular corpus.
• Cost: A credibility-aware interface can supply $a=\mathrm{LLM}(q,\{(d_i,c_i)\}_{i=1}^k)$, but producing and validating $c_i$ is a separate expense and signal-quality problem. Established reporting, peer review, publication date and citation history are possible features, not definitions of truth. An old source can be the right historical reference; a new article, encyclopedia page or model-generated text can be right or wrong. Score the source, claim, time and task with an explicit rubric rather than assigning universal tiers from age or writing style alone. A low-quality document can mislead a generator, but the effect is model- and setup-dependent. A vanilla transformer does not give all documents equal attention by default: positions, content and learned attention vary. The missing element is an explicit, validated credibility control, not a proof of uniform weighting. In CrAM’s ideal-setting Llama2-13B results, adding one generated misinformation document to four retrieved documents changes NQ EM from $28.9\%$ to $11.9\%$ (seventeen points) and TriviaQA EM from $62.5\%$ to $28.0\%$ ($34.5$ points). This already rules out a universal twenty-to-thirty-point law.[2]

Contamination experiment

• What: Stipulate a $1{,}000$-question medical toy, $k=5$ documents and $82\%$ baseline correctness; do not present it as a replicated clinical benchmark.
• Why: Applying an assumed twenty-to-thirty-point loss illustrates scale, not a measured effect transferable from another task.
• Result: Candidate accuracy would be $62\%$ to $52\%$, or $620$ to $520$ correct answers versus $820$ initially.
• Limit: This gives $200$ to $300$ fewer correct answers in net. It does not establish which or how many answers changed without paired outputs. If no initially incorrect answer improves, the loss would equal $200$–$300$ correct-to-incorrect flips: $20\%$–$30\%$ of all questions, or approximately $24.390244\%$–$36.585366\%$ of the initially correct answers. If some answers improve, there can be more losses and more total flips for the same net decline. “One in four to one in three flips” is not determined by the aggregate rates. Keeping the top three of five can retain a contaminant and discard a good anchor. It can also help in some workloads; evaluate matched evidence changes rather than claiming that a smaller context always fails or solves credibility.

Operating rules

• What: Preserve credibility metadata, provenance and score versions where meaningful; evaluate single-contaminant and correlated-contaminant stress cases.
• Why: Explicit reliability information targets a dimension relevance alone does not guarantee.
• Limit: An index-time score can become stale and may be query-dependent. It is not automatically negligible or reusable without refresh.
• Cost: Separate offline scoring, updates, storage and online scoring/lookup costs. Benchmark them rather than calling one additional score free. Prioritize source-quality controls in consequential domains, with independent validation and escalation when evidence conflicts. A homogeneous corpus does not establish uniformly correct, current or attack-free content; do not remove the axis solely because all documents share a publisher.

35.3 RA-RAG: reliability from cross-source agreement

Offline reliability estimation

• What: RA-RAG means Reliability-Aware RAG. It estimates source agreement using source-specific generated answers, filtered for alignment with their own retrieved documents.
• Why: Source IDs keep document volume from directly multiplying ballots within one identity.
• Limit: A score estimates agreement on covered calibration questions, not independently known factual reliability. The original $200$-query calibration is an experiment size, not a universal requirement.
• Cost: Generate/filter calibration answers, iterate vote-weight updates and declare stopping/tie rules. Convergence and correct truth recovery are not guaranteed merely by repeating the updates.[3] Let $A_s(q)$ be source $s$’s filtered answer or abstention $\mathrm{IDK}$, and let $\mathcal Q_{\mathrm{res}}$ be the calibration queries with a resolved non-abstaining consensus answer class $z(q)$. Among the $Q$ calibration queries, define $J_s=\{q\in\mathcal Q_{\mathrm{res}}:A_s(q)\ne\mathrm{IDK}\}$. The bounded agreement estimator is:

Here $\sim$ means the chosen answer-equivalence rule and $S$ is the source count. The paper excludes abstentions from the reliability denominator; the chapter’s earlier division by $Q$ is valid only when every calibration query yields a usable source answer. Handle unresolved/all-abstaining queries explicitly rather than counting matching “IDK” strings as factual agreement, and mark zero-answer sources unknown. The paper rescales agreement into vote weights $v_s=S r_s-1$ (with a possible cap on the scaling factor); $r_s\in[0,1]$ but $v_s$ need not lie in that interval or be nonnegative. This is an affine vote weight, not a normalized probability.[3] A stable consensus may still be wrong: copied false answers, correlated sources, a shared generator or filter errors can create a self-reinforcing fixed point. Use independent labeled audits, answer-coverage counts, domain/time slices and source-dependence checks before interpreting agreement as truth reliability.

Online per-source voting

• What: Inspect sources in descending estimated reliability. Retrieve and generate from each source separately, filter unsupported answers to abstention, and stop after $K$ usable sources or exhaustion.
• Why: The original $\kappa$-RRSS procedure checks relevance through the source answer/filter, not a free pre-generation relevance oracle.
• Vote: For a selected source set $\mathcal W$ and answer classes $\mathcal A$, use $\hat a\in\mathop{\mathrm{arg\,max}}_{a\in\mathcal A}\sum_{s\in\mathcal W}v_s\mathbf1[A_s(q)\sim a]$, with declared tie/empty policies.
• Cost: $K$ is the accepted-source target, not an upper bound on source-model calls. With many abstentions, all $S$ sources can be attempted. Later RA-RAG adds semantic-equivalence clustering; do not silently mix versions or count paraphrases as separate truths.[3][4] In the constructed COVID-19 vote, three named sources support SARS-CoV-2 with assumed agreement rates $0.95$, $0.93$, $0.90$; three others support the false 5G claim with $0.20$, $0.15$, $0.15$. These are toy values, not measured publisher ratings. Raw nonnegative rate-weighted totals are $2.78$ versus $0.50$, breaking a $3{:}3$ unweighted tie. That is a simplified vote, not the paper’s transformed weights: with $S=6$, the corresponding sums of $6r_s-1$ are $13.68$ versus $0$. Do not interpret either score or its normalized share as a calibrated answer probability. One ballot per identity limits duplication inside that identity, not attacks using many identities or independent-looking copies.

Cost and measured results

• What: For a budget illustration, $S=1{,}000$, $k=5$ and $K=4$ give $5{,}000$ documents across all sources versus twenty documents attached to four accepted source calls.
• Arithmetic: At exactly $500$ input tokens per document, these portions use $2.5\times10^6$ versus $10{,}000$ document-input tokens: a $250{:}1$ ratio and $1-20/5{,}000=99.6\%$ reduction.
• Limit: These are not one giant generator context versus one twenty-document context in the actual algorithm, nor total measured tokens/cost. Rejected sources, repeated query/instruction tokens, outputs, filtering and selection work are omitted.
• Offline cost: $S Q=200{,}000$ source-query answers at $Q=200$ before scoring/iteration overhead. Cached answers can be reused across weight iterations; do not multiply them by the iteration count unless regenerated. For a fully specified published cross-check, the later RA-RAG revision’s NQ/GPT-4o-mini experiment with $1{,}000$ sources and $K=4$ reports $627{,}115$ versus $5{,}415$ total processed tokens per query, without versus with RRSS: approximately $99.136522\%$ fewer. Mean API calls are $1{,}000$ versus $8.66$, not four; EM changes $0.780\to0.768$. This comparison is selection versus all-source aggregation, not standard RAG. The draft’s unscoped $73.7\%/63.4\%$ and $91.3\%/81.2\%$ pairs cannot establish a matched cross-method ranking and are not used as such here.[4] Consider RA-RAG when source identity is meaningful and calibration coverage is sufficient. Choose the source budget by measured cost, evidence coverage and error—not a universal calibration size. Re-estimate under domain, ownership, copying or behavior changes; record answer-equivalence/filter versions and independently check correctness, not only agreement stability.

35.4 CAG: training the model to reason about credibility

Multi-granularity training

• What: CAG constructs document- and sentence-granularity credibility-annotated training inputs, then fine-tunes a generator to use those annotations.
• Why: It can learn to use credibility on unfamiliar documents without requiring the same source-ID reputation table as RA-RAG.
• Limit: It does not eliminate credibility assessment. The original training construction bins retrieval relevance into high/medium/low levels; its benchmark additionally incorporates timeliness and source reliability. Those proxies are not an independently verified truth oracle.
• Cost: Annotation, teacher generation, training, validation and inference-time credibility inputs all require resources; “days” is a project estimate, not a method guarantee.[5] The original teacher is GPT-3.5. It receives the question, credibility-tagged documents and the gold answer, then writes a credibility-guided explanation used as the training output. The model learns to generate with annotated evidence; it is not simply trained to infer source truth from writing register or citation appearance. A coherent, gold-conditioned explanation is also not a verified causal trace of the student’s internal computation.[5] As one explicit benchmark policy, relevance tier $R\in\{1,2,3\}$ is downgraded by elapsed time $T\ge0$ in intervals $\tau\gt 0$, and capped by source tier $S_{\mathrm{tier}}\in\{1,2,3\}$:

This is a declared ordinal heuristic, not a calibrated probability. Its choice of timestamp and source tier still needs validation.[5]

Why prompting alone loses

• What: An untrained baseline can receive the same credibility labels in its prompt.
• Why: It is a useful matched control for the value of fine-tuning.
• Limit: Prompted labels can help, harm or do little; they do not universally “remain ignored.” In the CAG table, ChatGPT HotpotQA EM moves $0.334\to0.422$ when credibility prefixes are added, while other model/task pairs behave differently.
• Cost: Fine-tuning adds offline work; deployment still needs valid annotations and their acquisition cost. Do not call inference content-only or zero-lookup solely because weights were trained.[5] In the illustrative Nobel conflict, an official record identifies Wilhelm Röntgen as the first Nobel Prize in Physics recipient in 1901; a blog incorrectly names Einstein. The credibility-tagged example teaches how to prefer the better-supported source. It does not prove all official-looking pages are correct or that a high label should override direct contrary evidence without review.

Reported experiments and operating rules

• What: Original Table 2 reports CAG-7B HotpotQA EM $0.509$ versus retrieval-based LLaMA-2-7B $0.280$.
• Arithmetic: The absolute gap is $22.9$ percentage points; relative improvement is $(0.509/0.280-1)100\%\approx81.785714\%$, rounded to $82\%$.
• Metric: The paper’s short-answer EM protocol checks whether a gold answer is an exact substring of the generation; it is not a universal clinical accuracy or token-overlap $F_1$ definition.
• Noise condition: At RGB noise ratio $0.8$, the same table gives CAG-7B $0.897$ versus retrieval-based ChatGPT $0.773$, a $12.4$-point gap—not $89.0\%$ and $11.7$ points. Noise here includes distracting context; it does not mean every noisy item is synthetic falsehood.[5] Solving $50.9/1.82\approx27.967033$ percent recovers an approximate baseline only if the rounded $82\%$ is treated as exact. Use the actual reported $28.0\%$ baseline for precise comparisons. The $22.9$-point absolute gap and approximately $82\%$ relative improvement are different denominators. RGB compares different base systems; it does not isolate training alone or establish a universal ranking against RA-RAG or CrAM. Consider CAG when annotation quality, inference-time signals, training access and validation justify the expense. Sentence granularity can expose local conflicts, but its cost and error rate must be measured. Audit label inversion and distribution shift, and compare alternatives on matched tasks rather than assuming fine-tuning is always broader or better.

35.5 CrAM: down-weighting attention on gullible heads

Causal head selection

• What: CrAM means Credibility-aware Attention Modification. On known misinformation examples, it modifies one head at a time using document-credibility weights, then measures the change in a known wrong answer’s generation probability.
• Effect: $\mathrm{IE}_h=P_0(a_{\mathrm{wrong}})-P_{h}^{*}(a_{\mathrm{wrong}})$. A positive mean effect means this particular modification reduced that wrong answer’s probability on calibration data; it is not head removal or proof that a head only encodes misinformation.
• Calibration: The paper uses one hundred examples per dataset to rank heads and another hundred for validation of how many to select. This is labeled misinformation calibration even though no gradient update is used.
• Cost: Model internals, probability evaluation, intervention implementation and held-out validation are required. The toy one-hundred-to-three-hundred-head range below is not a universal setting.[2] Head effects depend on the model, prompt, task, intervention and score quality. Individually measured effects need not add when many heads are changed. Recalibrate after material model/domain changes and test normal-capability regressions as well as attack cases.

Inference-time intervention

• What: Reweight selected heads’ post-softmax attention to key-token positions, then normalize each query-token row.
• Why: This directly changes use of document tokens instead of merely adding a textual instruction.
• Limit: Attention weights are not causal explanations by themselves, and the intervention can suppress correct evidence if credibility is wrong.
• Cost: It requires access to the attention implementation, including compatible masking/caching/kernels. A hosted text-only interface cannot perform it just because no fine-tuning is needed.[2] For one head $h$, query-token row $i$ and key token $j$, let $\alpha_{h,i,j}\ge0$ be normalized attention on allowed positions. Let $c_j\in[0,1]$ be an already prepared token multiplier, with non-document tokens normally assigned one. CrAM’s paper min-max scales raw document scores $s_d$ using $(s_d-s_{\min})/(s_{\max}-s_{\min})$ when the range is positive. Declare a policy for equal raw scores (zero range) rather than divide by zero. The row-wise operation is:[2]

Here $\mathcal A_i$ is the allowed key set for row $i$. Masked positions remain zero, and unselected heads retain their original normalized row. If $Z_{h,i}=0$, use an explicitly chosen safe fallback, such as preserving the original valid row and escalating, rather than silently creating NaNs. This fallback and an equal-score policy are implementation requirements, not additional guarantees from the paper. Multiplying post-softmax weights is equivalent to adding $\log c_j$ to unmasked logits when $c_j\gt 0$; adding raw $c_j$ to logits is a different operation.

Attention arithmetic and measured scope

• What: For a generic reweighting illustration, Document A has initial row mass $0.30$ and multiplier $0.8$; B has mass $0.20$ and multiplier $0.1$. The remaining $0.50$ is on unscaled tokens.
• Arithmetic: Scaled masses are $0.24$, $0.02$ and $0.50$, so the full denominator is $0.76$. Full-row shares become $0.24/0.76\approx31.578947\%$ for A and $0.02/0.76\approx2.631579\%$ for B.
• Conditional share: Among A and B only, A moves from $0.30/0.50=60\%$ to $0.24/0.26=12/13\approx92.307692\%$; the A:B odds are $12{:}1$. The latter percentage is not A’s total attention share.
• Normalization scope: These are supplied multipliers, not a literal two-raw-score run of the paper’s min-max transform. If the only raw document scores were $0.8$ and $0.1$, they would map to one and zero instead. A toy thirty-two-layer model with thirty-two heads per layer has $1{,}024$ heads. Selecting one hundred to three hundred changes approximately $9.765625\%$–$29.296875\%$ of heads; $70.703125\%$–$90.234375\%$ remain unselected. This counts heads, not the fraction of FLOPs, parameters, tokens or total model behavior changed. It is not Llama2-13B’s forty-layer, forty-head configuration. For a scoped result, CrAM’s ideal-score setting with four retrieved documents plus one generated misinformation document reports Llama2-13B NQ EM $33.6\%$; Llama3-8B achieves NQ EM $36.9\%$ and TriviaQA EM $64.4\%$. With GPT-generated scores, the Llama3-8B values are $30.7\%$ and $62.2\%$. These are QA measurements under a controlled injection setup, not a verified $91.3\%$ universal attack-defense or near-oracle rate. A different RA-RAG paper’s NQ number cannot explain this gap causally by “narrow head coverage.”[2] Consider CrAM when no weight update is desired but scorer quality, model access and calibration support the intervention. Restrict changes to a validated set, monitor shifted attacks and clean performance, and benchmark end-to-end overhead. “Plug-and-play” does not imply universally safe, inexpensive or same-day deployment.

35.6 Choosing among the three

Two resource gates

• What: Use resource checks to form a feasible candidate set: source identities and coverage for RA-RAG; credible annotations plus training for CAG; scores, labeled misinformation calibration and attention access for CrAM.
• Why: Missing inputs cannot be repaired by choosing a method’s largest headline number.
• Limit: Stable identities do not automatically make RA-RAG best, and lack of a labeled training corpus does not make CrAM feasible.
• Cost: Time estimates are project-specific. Compare matched quality, safety, scoring, serving and maintenance costs after checking prerequisites. The three methods can compose, but compatibility is not free calibration. RA-RAG’s source agreement rate may be an input feature for an upstream document scorer; it is not automatically the right query-specific CrAM token multiplier. CAG also needs credibility inputs at inference. Include a “prerequisites missing: obtain evidence, restrict scope or defer” branch instead of treating CrAM as an unconditional residual leaf.

Per-pool marketplace example

• Pool 1: Identified seller listings and regulator notices make source-aware calibration possible, but verify identity quality and source dependence before choosing RA-RAG.
• Pool 2: Anonymous reviews with usable annotated spam examples may support CAG; spam labels must be mapped and validated for the actual credibility task, including inference inputs.
• Pool 3: A same-week partner launch with neither resource can consider CrAM only if scores, known misinformation examples and internal model access already exist.
• Limit: A deadline does not establish scorer correctness, calibration coverage or deployment readiness. Otherwise restrict the feature or obtain the missing resources. Evaluate per content pool and domain. A shared interface should preserve score semantics, provenance, coverage, uncertainty and timestamps—not merge all three methods’ outputs into a supposedly interchangeable probability.

Timed CrAM calibration

• What: Stipulate one hundred calibration examples, $1{,}024$ heads and a measured-equivalent service-time assumption of $50\,\mathrm{ms}$ per single-example probability-evaluation pass; this is not a hardware benchmark.
• Arithmetic: There are $100\times1{,}024=102{,}400$ head-modified passes. Their assumed serial service time is $5{,}120\,\mathrm{s}$, or $85.333333$ minutes.
• Baseline: Reusing one unmodified baseline pass per example adds one hundred passes: $102{,}500$ total and $5{,}125\,\mathrm{s}$, or $85.416667$ minutes.
• Limit: Additional validation, head-count search, scoring, setup, variable answer/context lengths and integration are omitted. Batching changes what “one pass” means; do not combine a batched-pass time with a single-example pass count. The baseline-inclusive toy is under $1.5$ hours of serial service time under its assumptions, not proof that calibration and deployment finish in an afternoon. At $S=1{,}000$ and $Q=200$, RA-RAG instead needs up to $200{,}000$ calibration source-query answers before other work. CAG includes annotation and training. These quantities do not determine engineer time or invoices; larger corpora can also change retrieval, coverage and calibration needs. Reassess feasibility when models, sources, scorers or resources change. Plan maintenance rather than an automatic expiration date for one method, and escalate low-confidence or conflicting credibility evidence to an appropriate review workflow.

35.7 Temporal awareness and credibility laundering

The citation-chain blind spot

• What: A claim has observation or substantive-verification time $t_0$, then is repeated on pages $C_1,\ldots,C_n$ with publication dates $t_1,\ldots,t_n$.
• Why: A recency proxy based on $t_n$ can overstate recency of the underlying measurement.
• Limit: The earliest publication is not always the right validity date: later independent measurement or verification can legitimately refresh a time-sensitive claim, while a historical fact need not decay.
• Cost: Trace the reachable provenance graph or store versioned claim-level dates offline. Work depends on graph depth, branching, caching and query volume, not necessarily a walk over the entire corpus. A statistic measured in 2020 and copied in 2023 has not been remeasured merely because a page is new. Repeated wrappers can evade a publication-age filter, but their new dates must be distinguished from retrieval/access dates and genuine new measurements. A reference-list omission can also propagate downstream. It is a coverage/provenance concern, not evidence that every later author independently checked completeness or that every missing reference is deliberately suppressed.

Freshness-weight experiment

• What: Use $w(T,t)=2^{-(T-t)/\tau_{1/2}}$ for evaluation time $T\ge t$ and half-life $\tau_{1/2}\gt 0$ in the same units.
• Assumption: Treat the calendar-year labels as exact year offsets, with $T=2026$ and $\tau_{1/2}=1$ year; this is not day-accurate timestamp arithmetic.
• Arithmetic: The 2023 wrapper yields $2^{-3}=1/8=0.125$, while the 2020 observation yields $2^{-6}=1/64=0.015625$. The ratio is exactly eight.
• Limit: This is a freshness-weight distortion under an illustrative decay policy, not an eightfold probability of being wrong. A 2025 wrapper gives $1/2$ and a ratio of $32$ to the unchanged observation weight. Generally,

At this fixed $T$, $t_0$ and half-life, the maximum ratio is $64$, not unbounded as chain length grows. A date advance $\delta$ contributes a factor $2^{\delta/\tau_{1/2}}$; hop factors telescope to the total date advance. More hops with the same final timestamp do not increase distortion. Unbounded ratios require allowing the elapsed origin-to-current time to grow, not merely adding citations.

Operating rules and claim limits

• What: Track measurement/validity dates separately from publication, modification, citation and retrieval dates; distinguish time-persistent from time-sensitive claims.
• Why: A reliable source can repeat old evidence, and a stale label can hide an independent update.
• Limit: Source reliability alone does not recover the observation date. CAG can incorporate timeliness when it is supplied correctly; none of these methods guarantees its provenance is correct.
• Cost: When origin/verification time is unknown, report uncertainty and avoid a fabricated freshness boost. Revisit claims according to domain change rates rather than one universal half-life. Treat frequent re-citation as a provenance audit signal, not automatic consensus or automatic laundering. Three independent measurements differ from three copies of one observation. Keep publication and modification fields, but neither replaces a claim’s observation/validity record. Offline annotations can reduce online work only if provenance, updates and any contractual audit requirements are actually maintained.

35.8 Pluralism: when every credible source shares a blind spot

Credibility-only ranking misses ambiguity

• What: A position-weighted utility ranker selects a permutation maximizing $\sum_{k=1}^{K}u(d_{\pi(k)})v(k)$ with nonnegative, nonincreasing attention discounts $v(k)$.
• Why: This accounts for relevance/credibility utility and position, but has no explicit intent-coverage or redundancy term.
• Limit: If a candidate pool lacks a relevant intent, reranking that pool cannot invent its missing evidence. Once intent is explicit, unrelated entities should not be rewarded merely for difference.
• Cost: Define utility and query intent first; diversify only over eligible, sufficiently relevant evidence. Position discounts are often modeled in an NDCG-like form, but measured user attention is not automatically identical to an evaluation metric’s chosen discount. Changing discount scale changes objective and exposure units. On the ambiguous query “Michael Jordan,” several entities can be plausible. If the researcher is the known intended entity, a basketball-dominated list is a relevance/intent-recognition failure. The toy below instead models coverage of four plausible intents before clarification; its utility values are stipulated ranking scores, not known user-intent probabilities or measured truth.

ILAD, coverage, and the combined objective

• What: For $K=|L|\ge2$ and nonzero embeddings, define $\mathrm{ILAD}(L)=\frac{2}{K(K-1)}\sum_{i\lt j}(1-\cos(e_i,e_j))$.
• Why: It summarizes embedding dissimilarity, not necessarily factual independence or meaningful stance coverage.
• Limit: Cosine dissimilarity lies in $[0,2]$, not automatically $[0,1]$; unrelated or adversarial text can increase it. It is not a probability or, without further conditions, a metric distance.
• Cost: Specify embeddings, normalization and handling of $K\lt 2$ (for example, omit the term). Geometric spread needs an independently defined task-quality check. Let $\mathcal S$ be a fixed, nonempty reference set of eligible intents/stances and require $S_d\subseteq\mathcal S$. Then coverage is bounded in $[0,1]$:

For a fixed $K\ge2$ and declared $\lambda_1,\lambda_2\ge0$, one set-level objective is:

At $\lambda_1=\lambda_2=0$, this recovers a top-$K$ utility set, with tie handling; it does not literally restore the earlier position-weighted formula. For nonnegative decreasing discounts, sorting that set by utility maximizes the separable position-weighted version. The utility sum scales with $K$ while the two averages do not, so weights require retuning when list length or score scale changes. Carbonell and Goldstein’s MMR is a relevance–redundancy heuristic; it can be useful without an enumerable stance set but does not certify coverage of one.

Michael Jordan worked example

• Candidates: Twenty documents represent fourteen basketball, three footballer, two actor and one researcher items. Specify basketball utilities $0.90,0.89,0.88,0.87,0.86$ and nine more at $0.85$; footballer utilities $0.75,0.74,0.73$; actor utilities $0.70,0.69$; researcher utility $0.62$. The leading values are not group means.
• Geometry: Stipulate every within-intent cosine dissimilarity as $0.08$ and every cross-intent dissimilarity as $0.75$; these similarities admit a valid Gram matrix.
• Pure utility: At $K=5$, the five largest scores sum to $4.40$, with coverage $1/4$ and ILAD $0.08$.
• Full-coverage candidate: Take the top two basketball items plus the best item in each other intent. Its utility is $0.90+0.89+0.75+0.70+0.62=3.86$, not $3.85$. The loss relative to $4.40$ is $0.54/4.40\approx12.272727\%$, not $12.5\%$. The full-coverage candidate has ten pairs: one within-intent and nine cross-intent. Thus $\mathrm{ILAD}=(0.08+9\times0.75)/10=0.683$ exactly, an $8.5375$-fold increase over $0.08$. Importantly, the draft’s weights $\lambda_1=0.3$, $\lambda_2=0.5$ do not select that full-coverage optimum. Its objective is $3.86+0.3(0.683)+0.5=4.5649$. Three basketball items plus footballer and actor instead give utility $4.12$, ILAD $0.549$, coverage $0.75$, and objective $4.6597$. Exhaustively checking all $\binom{20}{5}=15{,}504$ sets confirms the latter maximum. For the corrected full-coverage illustration in Figure $35.8$, hold $\lambda_1=0.3$ and set $\lambda_2=1$. Full coverage then maximizes this fully specified toy objective at $5.0649$. The all-four-intent candidate becomes optimal at $\lambda_2=0.8792$ (tied at the threshold, unique intent-count pattern above it). This is a toy trade-off, not a universal coverage weight. Search-diversification literature motivates balancing utility and coverage, not a universal acceptable cost. A sixty-percent utility loss for a twenty-five-point coverage gain warrants checking intent, eligibility and scaling, but it is not mathematically invalid if an explicit policy values that coverage sufficiently. Use coverage where the task has multiple eligible intents or viewpoints; clarify or route unambiguous queries rather than manufacturing balance. The draft’s $95\%$ unambiguous-query share and $15\%$–$20\%$ utility-loss cap are workload/policy hypotheses, not measured universal constants. Build and audit the intent set, missing labels and source independence; do not equate geometric separation with credible pluralism. Choose weights or constraints against matched quality and harm budgets.

35.9 Fair exposure: doubly stochastic rankings and group constraints

From a list to an exposure policy

• What: For $N$ candidates and $K\le N$ displayed positions, let $P_{d,k}$ be the probability that document $d$ occupies position $k$.
• Exposure: $e_d=\sum_{k=1}^{K}P_{d,k}v(k)$, with $v(k)\ge0$; for a nonempty group, $E_G=|G|^{-1}\sum_{d\in G}e_d$ is average per-document exposure.
• Limit: Group-total exposure is $|G|E_G$, a different quantity. A hard ranking gives zero displayed exposure below the cutoff; a randomized policy can spread it, but not create additional attention mass.
• Constraints: Use nonnegative entries, unit column sums and row sums at most one for a rectangular displayed-position matrix. The correct rectangular feasibility conditions are

Their total mass is $K$. Requiring all $N$ row sums and all $K$ column sums to equal one is impossible when $K\lt N$. For the standard square doubly stochastic formulation, add $N-K$ unshown positions with zero attention and complete each row to one. If $K\lt N$, one explicit completion gives every extra column entry $(1-\sum_{k\le K}P_{d,k})/(N-K)$; those extra columns sum to one. When $K=N$, the rectangular constraints already force every row sum to one. Apply Birkhoff to this square completion, not directly to an incompatible rectangular matrix.[6] For nonempty groups with positive average utility $\bar u_G$, demographic parity uses $E_{G_1}=E_{G_2}$, while the paper’s utility-proportional (“disparate treatment”) constraint uses $E_{G_1}/\bar u_{G_1}=E_{G_2}/\bar u_{G_2}$. Here “disparate treatment” names that mathematical formulation, not a general legal conclusion. Fix utilities first so this is linear in $P$. Zero/negative utilities need an explicitly defined alternative; do not divide by zero. Choosing parity, merit-proportional exposure or another rule is a policy decision informed by validity/bias of the utility estimates—not a theorem that credibility mandates one rule.[6] One LP maximizes $\sum_d u(d)e_d$ under these assignment and declared linear group constraints. Its coefficients require a fixed query, utility model, group definition and attention model. It allocates modeled expected utility, not necessarily end-to-end answer accuracy. Check feasibility: exposure targets outside the candidate/position polytope cannot be met by merely renormalizing totals. Singh and Joachims formalize the exposure LP; amortized fairness across a changing query stream is a further accounting choice.[6]

Serving and approximation

• What: Decompose a square doubly stochastic completion as $\bar P=\sum_\ell\alpha_\ell\Pi_\ell$, where $\alpha_\ell\ge0$, $\sum_\ell\alpha_\ell=1$ and each $\Pi_\ell$ is a permutation matrix. Sample one permutation and display its first $K$ items.
• Why: The displayed-position marginals reproduce $P$ and the specified exposure in expectation.
• Limit: Individual sampled lists need not meet the group target, and a finite query stream can deviate. Track query-specific eligibility, target mass and realized exposure; expectation is not a per-query floor.
• Alternative: FA*IR constructs rankings with protected-representation conditions on prefixes, using a binomial-test-derived minimum-count schedule. That is different from utility-proportional expected exposure; neither has a universal greater accuracy cost than the other.[6][7] LP serving feasibility depends on candidate size, constraints, solver, caching and latency budget—not QPS alone. The chapter’s six graph/frontier policies are heuristic candidate-allocation controls: two alter graph links/group domination; four steer which neighbors or batches are admitted. They do not directly optimize position-discounted exposure or inherit an LP guarantee. Validate both final displayed exposure and retrieval/answer quality. Offline optimization is reusable only while the relevant candidates, utilities, attention model and constraints remain applicable.

Exposure worked example

• What: Twelve documents form disjoint groups of eight, three and one. For this toy, every item—not merely the group mean—has utility $0.90$, $0.75$ and $0.60$ in its respective group.
• Discounts: Use the stipulated rounded vector $(1.000,0.631,0.500,0.431,0.387,0.356)$ as exact toy inputs, so $V=3.305$. Do not call it an unrounded empirical attention curve.
• Hard ranking: All six highest-utility slots go to Group A; average exposure is $3.305/8=0.413125$ for A and zero for B/C. The positive-to-zero ratio is undefined in ordinary real arithmetic (infinite only by an explicit extended convention), not the target utility ratio $1.2$.
• Target: With group averages $a=0.90t$, $b=0.75t$, $c=0.60t$, conservation gives $(8(0.90)+3(0.75)+0.60)t=10.05t=3.305$. Thus $t\approx0.328855721$, $a\approx0.295970149$, $b\approx0.246641791$ and $c\approx0.197313433$. Group A’s average exposure falls by $1-0.295970149\ldots/0.413125\approx28.358209\%$. The targets are feasible here, not merely mass-balanced: set $P_{d,k}=u(d)/10.05$ for every displayed position. Each column sums to one and each row sum is $6u(d)/10.05\le1$; then $e_d=3.305u(d)/10.05$ realizes the targets. Completing this matrix to $12\times12$ and decomposing it constructs valid sampled lists. With homogeneous within-group utilities, expected utility moves from $2.9745$ to approximately $2.804317164$; no inference about answer accuracy follows from that utility difference. Choose the fairness definition with stakeholders and audit the utility/credibility estimates and eligible groups. Use expected-exposure sampling when that guarantee fits the requirement; use an appropriate per-list constraint when it does not. Monitor finite-run exposure and changing query mixes. A graph approximation must be validated against the actual target, and “already fair” under one chosen rule does not establish fairness under every other rule.

Diagrams

Figure 35.1

Diagram description and source

flowchart LR
  r["Retrieval checkpoint"] --> g["Generation checkpoint"] --> c["Checking checkpoint"]
  r --> ra["Start: robustness and privacy"]
  g --> ga["Start: fairness and transparency"]
  c --> ca["Start: factuality and accountability"]
  ra -.-> x["Cross-stage responsibilities; investigate causes"]
  ga -.-> x
  ca -.-> x

Figure $35.1$: Six dimensions are tracked across the pipeline. The stage tags are operational starting points, not exclusive ownership or a causal diagnosis from one score. The survey’s privacy and robustness tests also evaluate the generator’s behavior.[1]

Figure 35.2

Diagram description and source

flowchart LR
  d1["Retrieved reference"] --> g["Generator without validated credibility control"]
  d2["Retrieved reference"] --> g
  d3["Retrieved reference"] --> g
  d4["On-topic misinformation document"] -.-> g
  d5["Retrieved reference"] --> g
  g --> a["Measure task- and model-specific degradation"]

Figure $35.2$: One misinformation document can degrade QA in a specified setup; it does not imply equal attention across documents or a universal twenty-to-thirty-point loss. Age alone does not make a source false. Match evidence, position and model settings when measuring the effect.[2]

Figure 35.3

Diagram description and source

flowchart TD
  o["Offline: filtered source answers and iterative agreement estimates"] --> w["Transform rates into vote weights"]
  w --> p["Try sources in descending reliability"]
  p --> g["Retrieve and generate separately for a source"]
  g --> f{"Filtered answer usable?"}
  f -- "no: abstain" --> n{"Sources remain?"}
  n -- "yes" --> p
  f -- "yes" --> k{"K usable sources reached?"}
  k -- "no" --> n
  k -- "yes" --> v["Aggregate answer-class votes with transformed weights"]
  n -- "no" --> v
  v --> a["Answer or declared empty/tie outcome"]

Figure $35.3$: RA-RAG’s accepted-source target $K$ does not bound attempted source calls. Alignment filtering, abstentions and answer equivalence matter; the bounded agreement rate and the paper’s transformed vote weight are distinct. Source duplication across identities and shared errors remain risks.[3]

Figure 35.4

Diagram description and source

flowchart TD
  a["Training: document- and sentence-granularity credibility annotations"] --> t["GPT-3.5 teacher: question, annotated documents and gold answer"]
  t --> f["Fine-tune generator on credibility-guided explanation outputs"]
  f --> i["Inference: question plus credibility-annotated evidence"]
  s["Validated upstream annotation policy"] --> a
  s --> i

Figure $35.4$: CAG learns to use credibility annotations at generation time. It is not inference from document content alone, and the teacher’s gold-conditioned explanation does not certify an internally faithful reasoning trace.[5]

Figure 35.5

Diagram description and source

flowchart TD
  c["Known wrong answers and misinformation calibration examples"] --> h["Rank heads by wrong-answer probability drop under one-head modification"]
  h --> v["Separate validation chooses head budget"]
  v --> s["Inference: access selected heads and multiply key-token attention by prepared scores"]
  u["Upstream document scorer plus normalization policy"] --> s
  s --> r["Normalize each allowed query row; handle zero cases"]
  r --> q["Audit clean quality, shifted attacks and latency"]

Figure $35.5$: CrAM selects heads by the effect of a specific credibility-based modification, not by deleting each head. No gradient update is needed, but known misinformation examples, upstream scores, validation and attention-internal access are prerequisites.[2]

Figure 35.6

Diagram description and source

flowchart TD
  a{"Meaningful source IDs and covered calibration questions?"}
  a -- "yes" --> r["RA-RAG feasible candidate; validate dependence and costs"]
  a -- "no or unsuitable" --> b{"Usable annotations, inference signals and training resources?"}
  b -- "yes" --> c["CAG feasible candidate; validate labels and generalization"]
  b -- "no" --> e{"Scorer, misinformation calibration, validation and attention access?"}
  e -- "yes" --> d["CrAM feasible candidate; validate intervention and regressions"]
  e -- "no" --> x["Acquire prerequisites, restrict scope or defer"]

Figure $35.6$: Feasibility checks are not a proof of the best method or fastest deployment. The methods may compose or fail a prerequisite; a held-out, matched evaluation still chooses among feasible options.

Figure 35.7

Diagram description and source

flowchart LR
  t0["2020: underlying observation"] --> t1["2023: summary publication"] --> t2["2025: another wrapper"]
  t2 --> r["2026: retrieved or accessed; not a new measurement"]
  f["Record observation, verification, publication and access dates separately"] -.-> r
  x["Trace substantive provenance or report unknown age"] -.-> t0

Figure $35.7$: Wrapper publication can reduce apparent age only when the filter mistakes it for a new observation. Retrieval time is not publication or measurement time. At fixed 2026 evaluation time and a 2020 origin, the one-year-half-life distortion is bounded by $64$; more hops alone do not make it unbounded.

Figure 35.8

Figure $35.8$: Under the fully specified twenty-candidate toy, utility-only selection totals $4.40$ and covers one intent. The corrected full-coverage list totals $3.86$, loses approximately $12.272727\%$ of utility, and has ILAD $0.683$. It is optimal at the displayed revised coverage weight; the original $\lambda_2=0.5$ instead selects a three-intent list. Within each chosen set, this display is sorted by utility, not a separate solved position-dependent diversity objective.

Figure 35.9

Diagram description and source

flowchart TD
  i["Eligible candidate set I"] --> b["Batch scored by reranker S and appended to result R"]
  b --> n["Pull neighbors from offline corpus graph CG"]
  n --> f["Frontier F steered by declared group-aware heuristics"]
  f -- "repeat while budget remains" --> b
  b --> e["Measure final rank-discounted group exposure and answer quality"]
  subgraph K["Illustrative source groups"]
    w["Wire service"]
    r["Regional outlet"]
    o["Independent outlet"]
  end

Figure $35.9$: The frontier loop steers candidate representation. It does not directly solve the exposure LP or guarantee parity/utility-proportional exposure after ranking. Compare the final displayed-policy measurements with the chosen target and monitor quality.

Whiteboard pack

What to draw

1. Draw retrieval, generation and checking; attach six dimensions as starting checkpoints, not exclusive ownership.
2. Show a radar chart with fixed axis order, labeled scale and $r_7=r_1$; do not use area as a safety certificate.
3. Draw on-topic good and bad evidence reaching a nonuniform-attention generator; label any degradation by its actual experiment.
4. Draw RA-RAG: source answers, alignment filtering, abstentions, conditional agreement, transformed vote weights and repeated source attempts.
5. Draw CAG: multi-granularity annotations, gold-conditioned teacher explanations, fine-tuning and annotated inference inputs.
6. Draw CrAM: known misinformation calibration, a separate validation set, selected head/row reweighting and upstream scores.
7. Write both $31.578947\%$ full-row A attention and $92.307692\%$ A-given-A-or-B share in the generic toy.
8. Draw a prerequisite branch that can end in “defer,” not an unconditional CrAM leaf.
9. Separate 2020 observation, 2023/2025 wrappers and 2026 access; write the bounded freshness ratio.
10. Draw the ambiguous-query toy with utility, ILAD and intent coverage; show why the original weights do not imply full coverage.
11. Write the rectangular ranking constraints: column sum one, row sum at most one.
12. Complete to a square zero-attention tail, decompose into permutations and sample one list.
13. Distinguish expected group exposure from each list and from graph-frontier candidate proportions; monitor realized exposure.

Spoken script

Credibility begins by separating signals and guarantees. Relevance is not truth, source agreement is not independent verification, and a new wrapper is not a new measurement. RA-RAG needs source-specific calibration and votes; CAG needs annotations both to train and to generate; CrAM needs scores, known misinformation examples and attention access. No method is automatically feasible or best when the others lack resources. On ambiguous tasks, explicit intent coverage can counter redundancy, but test the chosen objective and weights. For exposure, use correct assignment constraints and say whether fairness holds in expectation, on each list or over a monitored query stream. Keep source, date, label and evaluation provenance throughout.

Interview traps

1. A RAG system has a $2\%$ hallucination rate. Is it trustworthy?

No. Define the hallucination unit, reference and judge, then audit the other dimensions and relevant controls. Stage tags guide investigation, but privacy, robustness and factuality are cross-stage; a low observed rate does not prove absence of unauthorized data exposure or identify one cause.

2. Why does one bad document matter when nine others are credible?

A fluent conflicting document can steer a model, but vanilla attention is not uniform and the loss is not universally twenty to thirty points. Test the model, position, evidence set and task. A relevance threshold or smaller context may help or harm; neither is a validated credibility control by itself.

3. What makes RA-RAG resistant to a content farm?

One source identity receives one answer ballot, so duplicate documents inside that identity do not directly add votes. Correct reliability estimation also requires abstention-aware filtering and the declared vote transformation. Many identities, copied evidence, shared generators or an incorrect consensus can still defeat it; stable agreement is not proof of truth.

4. When does CAG beat CrAM, and when does CrAM win?

There is no universal winner. CAG trains a model to use credibility-annotated evidence, but still needs valid inference annotations. CrAM avoids gradient updates while requiring known misinformation calibration, validation, upstream scores and internal attention access. Compare both on matched tasks, clean regressions and total cost; cross-paper headline differences do not prove generalization mechanisms.

5. How do you choose among RA-RAG, CAG, and CrAM?

Check every prerequisite, not just source identity and training time. Source-aware calibration can make RA-RAG feasible; usable annotations and training can support CAG; scores, labeled calibration and attention access can support CrAM. If no method is feasible, obtain evidence, constrain the feature or defer. Evaluate freshness, intent coverage and exposure as separate requirements.

Key numbers

00-START-HERE · 01-READING-ROUTES · 34_evaluating_the_system · 36_provenance_and_adversarial_robustness