Appendix C: Design Checklists

00-START-HERE · 01-READING-ROUTES · A2_question_bank · A4_the_rag_card_and_index_datasheet Use these five operational checklists to turn a Retrieval-Augmented Generation (RAG) design review into named decisions, measured quantities, and explicit exit criteria.

TL;DR

• Run these checklists. Do not merely read them. Each line needs a number or a named decision.
• Internalize the retrieval checklist for interviews. It covers five decisions in ten minutes, with two minutes per phase.
• Run the indexing checklist before provisioning and before migrations. Derive memory, recall, latency, churn, and rebuild requirements.
• Run the evaluation checklist before trusting a dashboard. It must locate failures across retrieval, generation, and the full system.
• Run the credibility checklist before outsiders can write to the datastore. Trace sources, test attribution, limit attackers, and enforce access during retrieval.
• Run the latency checklist with a stopwatch on a real query. Measure five stages, optimize the dominant constraint, and derive cost per query.
• Treat every exit criterion as a gate. If the criteria fail, the design is not finished.

The story

Picture an aircraft crew preparing for departure. The crew does not admire five laminated cards and call the plane ready. They run every line. A box receives a measured value or a named choice. A box filled with “fast,” “large,” or “safe” remains open. The retrieval card is the route card. It asks how large the trip is, when to leave the gate, which airports to use, how to navigate, and how to brief the passengers. In system terms, it fixes problem size, the retrieval decision, the data sources, the search method, and the evidence shown to the generator. The capacity card counts source items, chunks, stored vectors, identifiers, links, text, replicas and rebuild artifacts separately. It distinguishes exact-neighbor recovery by ANN from human-judged task relevance; either can fail while the other looks good. The evaluation card calibrates the cockpit instruments. It checks the test set before flight, separates retrieval readings from generation readings, and traces a bad result to its owning stage. A single end-to-end score is like one warning light for the whole aircraft. It cannot tell the crew where the fault lives. The credibility and robustness card secures the cabin and cargo hold. It records where each document came from, what happens when sources conflict, whether citations support the answer, who can load content, and what one poisoned document can damage. It also enforces each passenger’s access rights during retrieval, before private material reaches the model. The latency and cost card starts the stopwatch and checks fuel use. It measures embedding, approximate search, reranking, prompt processing, and answer decoding under real load. The crew reduces the dominant delay first and records what quality each cheaper choice gives up. Departure requires five exit stamps. If any card lacks its stamp, the aircraft stays at the gate. The honest review says which requirement remains open. It does not hide the gap behind a confident adjective.

Decoder table

Core mechanics

These checklists are written to be run, not read. Each line must produce a concrete number or a named decision. A line that produces only an adjective remains unanswered. The retrieval checklist is the interview checklist to internalize. Reproducing its five-decision skeleton and sizing questions verbally in under two minutes is a reasonable preparation target. The other four checklists are job-side design-review instruments. Each checklist ends with exit criteria. If a design cannot meet them, report that the design is not finished.

C.1 The Retrieval Design Checklist

Run this checklist in ten minutes. It contains five decisions. Give one number or named choice for every line. A vague answer is a finding, not a pass.

Phase 0: Size the problem, 2 minutes

What

1. State source documents, chunks, vectors per chunk, dimension and precision; document count alone does not determine memory.
2. State sustained and peak QPS over named windows, query mix, concurrency and resource-service demand.
3. Define TTFT and total-latency endpoints and their required percentiles under load.
4. State source-to-serving freshness and deletion-visibility requirements, including allowable failure behavior. Minutes, hours and days constrain options but do not each dictate one architecture.
5. Define quality metrics, cutoffs, eligible populations and uncertainty; name mandatory constraints separately. Why This phase turns scale, load, responsiveness, freshness, and quality into testable requirements. Failure without it The design cannot size memory or compute. It also cannot prove that latency, freshness, or accuracy passes. Cost and complexity The source gives this phase two minutes. It separates two sizing axes, corpus size and query rate, and two latency measures, time-to-first-token and total.

Phase 1: When to retrieve, 2 minutes

What

1. Define $g(x)\in\{0,1\}$ explicitly, including an always-retrieve policy if chosen.
2. Name available gate signals and when they can be computed; benchmark their cost and paired benefit/loss slices.
3. Define the no-retrieval path, its permitted sources of knowledge, abstention and escalation rules.
4. Monitor gate risk, coverage and query drift. Distribution shift can degrade a gate, but degradation is not inevitable on every shift. Why The gate controls which queries pay retrieval cost and which queries rely on the closed-book path. Failure without it Retrieval becomes an unmeasured default. A shifted query mix can silently degrade a learned controller. Cost and complexity The source gives this phase two minutes. A learned gate also needs an ongoing drift-detection plan.

Phase 2: Where and what, 2 minutes

What

1. Enumerate the stores. If there is more than one, state the source-selection rule and what happens when two stores disagree. Source pointer: section $25.3$.
2. State query transformation $\Phi$. Choose identity, compression, expansion, rewriting, or decomposition. State which queries trigger it. Source pointer: chapter 24.
3. For conversational traffic, confirm that rewriting resolves referents outside the current turn. Source pointer: section $24.5$.
4. State the chunking scheme and its evidence. Semantic chunking needs a measurement on the target corpus, not a benchmark number. Source pointer: section $13.6$. Why This phase fixes the searched sources, transformed query, conversation behavior, and retrievable unit. Failure without it Multi-store disagreement has no policy. Rewrites can lose the subject, and chunking becomes an unsupported label. Cost and complexity The source gives this phase two minutes. It requires corpus-specific chunking evidence and query-specific transformation rules.

Phase 3: How to retrieve, 2 minutes

What

1. Name scorers and lexical/exact-identifier capabilities; justify dense-only choices on representative slices.
2. For hybrid retrieval, define normalization or rank fusion, constants, weights and missing-list behavior.
3. Separate retrieved candidates $k_0$, candidates actually scored and final context items $k$.
4. Name the index, exact-neighbor recall target and task-relevance target at a measured latency/load.
5. Define filters and fallbacks. Never relax ACL, tenant or mandatory policy constraints merely to recover recall. Why This phase makes ranking channels, depth, index behavior, and filtering reproducible. Failure without it The design can hide exact-match weaknesses, conflate three depths, or over-filter until no useful evidence remains. Cost and complexity The source gives this phase two minutes. Hybrid fusion adds a method and constant. Recall must be tied to a measured latency.

Phase 4: How to present it, 2 minutes

What

1. Count actual tokenized query, instructions, evidence, headers, separators and demonstrations, plus reserved output. $kL$ estimates evidence only for $k$ equal-length $L$-token chunks.
2. Evaluate evidence order and complete support-chain coverage on the deployed model.
3. Define serialization and instruction/data boundaries; formatting is not an injection-proof security boundary.
4. Record source IDs, versions and spans, whether citations are inline or post-hoc, and independently measured support and coverage. Why This phase connects retrieved candidates to prompt cost, evidence position, instruction boundaries, and citations. Failure without it Context cost stays hidden. Evidence can land in a weak position, instructions can blur with data, and citation timing remains ambiguous. Cost and complexity The two-minute phase is a practice allocation. $kL$ is an equal-chunk evidence estimate, not total prefill tokens; count the fully serialized prompt with the actual tokenizer.

C.1 exit criteria

What

1. Every one of the five decisions has an answer, and at least two carry a number.
2. Name one thing the design gives up without being prompted.
3. State how to find which stage broke when quality drops. Source pointer: chapter 34. Why These gates test completeness, trade-off awareness, and debuggability. Failure without it The retrieval design remains vague, one-sided, or impossible to diagnose. Cost and complexity The exit gate requires five answered decisions, at least two numeric decisions, one volunteered sacrifice, and one diagnostic plan.

C.2 The Indexing and Capacity Checklist

Run this checklist before provisioning and before every migration. Every line should produce a number. The source points to section A.8 for the arithmetic.

Sizing

What

1. Count $N$ stored vectors, dimension $d$, precision or packed code bytes, and per-item IDs.
2. Add only the selected index’s links, posting IDs, coarse centroids, PQ codebooks, maps and allocation/runtime overhead; do not attach every index family’s costs to every design.
3. State measured exact-neighbor and task-relevance effects of compression at matched search controls.
4. Give replication and rebuild overlap separately for indexes, payloads, metadata and caches. Twenty-percent reserve does not hold a complete second equal-size index.
5. Count source text and filterable metadata, compression and their own replication policy. There is no universal 30-fold payload crossover.
6. Derive per-partition fit and load. Ceiling division of aggregate bytes by usable RAM is a capacity lower bound, not a logical-shard or availability plan.[1] Why Sizing must include vectors, index overhead, compressed codes, rebuild copies, replicas, payloads, and usable node memory. Failure without it Fleet memory can be understated. Rebuilds can exceed capacity, and aggressive vector compression can hide a dominant payload store. Cost and complexity Budget actual simultaneous live and build artifacts. An old/new full-index overlap needs another full copy wherever both coexist; an external build also consumes resources. Replication is artifact specific, and payload dominance depends on representation and placement rather than a universal compression threshold.

Recall and latency

What

1. Define ANN Recall@$k$ against exact search on the same eligible corpus and metric. Report semantic/evidence recall separately with judgments.
2. Record search-time controls such as $\mathrm{efSearch}$ or $n_{\mathrm{probe}}$ and the measured quality/latency curve.
3. Record build/training settings and which changes require retraining, re-encoding, rebuilding or only configuration updates in this implementation.
4. Test realistic traffic, concurrency, warm/cold states, filters and failure capacity.
5. Prefer low-cost reversible experiments when feasible, but let measured failure cause and constraints determine the order of remedies.[2] Why This order tries reversible controls before expensive rebuilds or an index-family change. Failure without it Recall can be measured against the wrong target. Latency can look good only because the test ignored concurrency. Cost and complexity Search-time changes are often reversible; build-setting changes can require reconstruction. Neither a full rebuild for every change nor one mandatory optimization order is universal. Record the implementation’s actual migration and rollback path.

Churn and operations

What

1. State insert/update/delete rates and the implementation’s physical deletion or tombstone semantics.
2. Measure effects of deletions on recall, service demand, tail latency and memory; do not assume monotone recall loss or one universal fraction threshold.
3. Specify rebuild placement, full peak resources, transfer/swap consistency and what queries see throughout.
4. Test stale caches, replicas and failures while enforcing deletion and authorization state.
5. Pin document and query encoders, tokenizer, pooling, normalization and index versions. Document-space changes usually need re-embedding; query-only or verified compatible migrations may not. Versioned partial migration requires an explicit compatibility/routing plan.[3] Why The index is an operated system. Churn, deletion markers, rebuild behavior, and model versions change its live state. Failure without it Recall can decay without a trigger. Queries can hit an untested rebuild path, and a model change can invalidate the whole index unexpectedly. Cost and complexity Rebuild triggers use measured quality, latency, deletion visibility and capacity, not tombstone fraction alone. Some document-encoder changes require full re-encoding; query-only, compatible or dual-index migrations can permit other transitions.

C.2 exit criteria

What

1. Derive capacity by artifact and placement, including payload replicas, runtime and rebuild peaks.
2. Report graph/link and identifier shares under this layout; they do not shrink merely because vector codes shrink, but are not inherently incompressible.
3. Provide a timed rebuild/migration, deletion-verification and rollback runbook under target load. Why These gates demonstrate a scoped capacity plan, measured structural overhead and tested operations; they do not constitute a proof against every production failure. Failure without it Provisioning depends on an opaque estimate, and the rebuild path remains unmeasured. Cost and complexity Passing requires a derived capacity plan, structural-byte breakdown and timed operational runbooks. Explicitly label unknowns and unmeasured worst cases.

C.3 The Evaluation Checklist

An evaluation that cannot localize a failure is a dashboard, not an instrument. This checklist builds an instrument. The source develops the method in chapter 34.

Before measuring anything

What

1. Work three representative real queries by hand, including ambiguous and insufficient-evidence cases, before choosing metrics.
2. Describe sampling, query count, judges, relevance/gain scale, adjudication and agreement; keep development and test data separate.
3. Document training-contamination checks and their access limits. A closed model’s unknown corpus cannot be certified absent from a finite prompt probe.
4. Measure closed-book solvability, evidence dependence and required support. Closed-book-solvable questions can still test retrieval benefit or harm; they do not by themselves isolate retrieval quality.[4] Why This phase fixes the intended behavior, evaluation population, label process, and contamination checks before metric selection. Failure without it A metric can reward the wrong behavior. Contamination or closed-book solvability can make retrieval look useful when it was not tested. Cost and complexity The source requires three hand-worked real queries and a documented judgment set. It gives no fixed dataset size.

Retrieval layer

What

1. Report evidence Recall@$k_0$ and ranked relevance at final depth, with cutoffs, eligible corpus, gain, IDCG and unjudged/zero-IDCG policies.
2. Measure a fixed-generator oracle/gold-context reference on matched items. It estimates evidence headroom under that setup, not an absolute bound on all downstream or future improvements.
3. Validate model judges against human labels using classwise confusion, macro metrics, sample uncertainty and relevant slices; report the label process.[5] Why Separate candidate coverage, ordering, exact-neighbor approximation, evidence-use references and judge validity. No one metric substitutes for the others. Failure without it One retrieval metric can hide the other question. An unvalidated model judge can make the dashboard unreliable. Cost and complexity Budget exact-neighbor evaluation separately from human relevance judgments. Oracle-context and judge-validation runs need matched samples, explicit denominators and uncertainty.

Generation layer

What

1. Report context faithfulness and independent factuality separately; faithful use of a wrong source can still be false.
2. Define atomic claims and label support, contradiction, unverifiable and omitted claims without silently dropping unknowns.
3. Pair acceptance/abstention coverage with accepted-answer correctness and uncertainty, including zero-accepted cases.
4. Audit answer-length and claim-count effects. Use matched or stratified comparisons where needed; length dependence is a confound to address, not automatic invalidity of every comparison. Why These measures separate evidence use, truth, unsupported claims, coverage, and answer-length effects. Failure without it Wrong evidence can look faithful; refusals can inflate accepted-answer precision; answer-length and population shifts can confound headline comparisons unless controlled. Cost and complexity Claim-level scoring adds decomposition work. Conditional accuracy must be paired with abstention rate, and length must be checked across systems.

System layer

What

1. Classify a sample of failures across stages, allowing interacting and unresolved causes rather than forcing exactly one owner.
2. Define macro, micro and intermediate/slice aggregation and denominators.
3. Compare evidence removal, replacement and ordering under controlled decoding. Redundant evidence or distribution shift can complicate causal interpretation.
4. Run matched closed-book, oracle-context and shuffled-context controls with confidence intervals.
5. Measure production latency distributions, repeat-run stability, authorization failures and corrupted/poisoned-document robustness.[4] Why System-level controls connect quality to plausible mechanisms and operational constraints. An ablation or descriptive decomposition alone does not prove a unique cause. Failure without it A quality drop stays unlocalized. The system may ignore retrieval, and one average can hide unstable or fragile behavior. Cost and complexity The checklist adds three sanity cases, one ablation, repeated runs, percentile latency, and a corrupted-document test.

C.3 exit criteria

What

1. Set a triage SLO, such as one working day, for a two-point observed decline. First check uncertainty and label/traffic changes; permit an honestly unresolved cause.
2. Give every headline number its denominator, sample, version, uncertainty and missing-data policy.
3. Show matched closed-book and RAG results with evidence/quality/cost slices. Why These gates test localization speed, measurement transparency, and retrieval value. Failure without it The dashboard cannot assign a quality drop, explain its population, or show improvement over no retrieval. Cost and complexity One working day is an operational triage target for the illustrative two-point change, not a statistical detection guarantee or promise that every cause is identifiable.

C.4 The Credibility and Robustness Checklist

Retrieval places text the team did not write into a context the model treats as authoritative. This checklist covers unreliable sources, hostile sources, and sources that disagree. The source develops it across chapters 35 and 36.

Source quality

What

1. Record store, document/version and evidence span for each retrieved unit.
2. Define source reliability and authority policies, validation data and update ownership. Agreement and recency are signals, not proofs of truth; correlated sources can repeat one false claim.
3. Specify contradiction handling and whether to show competing evidence or abstain.
4. Preserve valid-time and ingestion/update-time metadata, plus the relevant jurisdiction or product version.
5. Trace quotations to original evidence where possible to detect credibility laundering. Why Source identity, evidence-based reliability, conflict handling, time validity, and laundering checks control what the model treats as authoritative. Failure without it Controls cannot be enforced by source. One contradiction, stale fact, or laundered claim can become a confident cited answer. Cost and complexity The checklist requires a reliability signal per source and a concrete contradiction mechanism. It gives no fixed numeric threshold.

Attribution

What

1. Define whether evidence preceded generation and whether citations were inline or post-hoc. Neither timing proves actual use, support or truth.
2. Resolve citation IDs/versions/spans and score attributable, extrapolatory, contradictory and unverifiable claims with explicit coverage denominators.
3. Validate automatic support judgments against human labels, with uncertainty and adversarial/ambiguous cases. Product grounding claims must match these measured properties.[6] Why Evidence availability, citation validity, support coverage and causal influence are different claims. Citation timing alone cannot establish or exclude all of them. Failure without it Product language can overstate post-hoc citations. An automatic score can look precise without known human agreement. Cost and complexity The source requires a human-scored sample for attribution and a human-agreement check for any automatic judge. It gives no fixed sample size.

Adversarial surface

What

1. Enumerate human and automated writers and upstream feeds as potential injection paths.
2. Separate instruction and data channels; add least-privilege tools, source authorization, sanitization where appropriate and verified output actions. A system-prompt sentence is not a security guarantee.
3. Measure harmful outcomes under one-chunk and coordinated attacks; state the tested threat model and unknown worst cases.
4. Authenticate ingestion and evaluate asset-bound credentials and signer trust; provenance credentials do not prove content truth, and absence does not prove fabrication.
5. Red-team retrieval, caches, tools and downstream actions, not only the base model. Why These checks cover write authority, instruction boundaries, single-document impact, provenance, and retrieval-specific attacks. Failure without it Untrusted content can act like an instruction. One poisoned chunk can have an unknown impact, and model-only tests can miss the retrieval path. Cost and complexity The source requires a measured one-chunk blast radius, authenticated ingestion policy, and a retrieval-specific red-team exercise.

Privacy and exposure

What

1. Enforce user and tenant entitlement before unauthorized evidence reaches the generator, including cache and reranking paths.
2. State the boundary mechanism, route validation, fail-closed behavior and tests. Neither a filter nor a separate index is inherently fail-open or fail-closed under every bug.
3. Specify deletion scope, measured visibility, replicas, caches, derived summaries/graphs, logs and backup handling; verify each required copy under policy.[7] Why The retrieval layer must stop unauthorized evidence before it reaches the generator and must remove every derived copy. Failure without it Private content can enter context. A filter bug can expose another tenant, and deletion can leave cached or derived material behind. Cost and complexity Separate stores may reduce cross-tenant risk but require correct routing and authorization. Either design needs tested failure behavior and a copy-aware removal policy.

C.4 exit criteria

What

1. State measured attacker outcomes and the threat model; do not call an untested worst case bounded.
2. Provide document removal with separate visibility, derived-copy, replica and retention deadlines plus verification evidence.
3. Keep grounding and safety claims within what evaluation and enforced controls demonstrate. Why These gates bound single-document harm, make deletion operational, and constrain product language. Failure without it The design cannot state its blast radius, complete removal, or defend its grounding claim. Cost and complexity Passing requires measured attack outcomes under a stated scope and verified removal operations; it does not establish universal worst-case security or instantaneous erasure of every copy.

C.5 The Latency and Cost Checklist

Run this checklist with a stopwatch on a real query rather than on a diagram. The source points to section $37.1$ and section A.7 for stage arithmetic.

Build the budget

What

1. Trace embedding, search, reranking, prefill and decode, plus gates, network, queues, retries and merge/orchestration work.
2. Report TTFT and end-to-end distributions at the relevant load and percentiles, including warm/cold behavior and errors. A median is useful but not a tail SLO.
3. Identify the actual critical-path and resource bottlenecks; do not assume generation or search always dominates.
4. Keep per-request stage traces: the sum of independently measured stage percentiles is not generally the end-to-end percentile. Why The budget exposes each stage, the production tail, the dominant constraint, and the product-facing latency measure. Failure without it An estimate can miss by an order of magnitude. A median can hide the production tail, and an optimization can target the wrong stage. Cost and complexity The source requires five timed stages at production concurrency. It calls for $p_{95}$ or $p_{99}$ when that percentile matters.

Reduce it in the right order

What

1. Compare context reduction with other remedies at matched evidence support and quality. Prefill includes linear model work and quadratic attention terms; total latency is not universally quadratic.
2. Reuse KV state only when token prefixes, model, positions, attention behavior and authorization are compatible. A repeated chunk in a different prefix is not automatically a reusable causal cache.
3. Test candidate/context cascades with their coverage and scheduling costs.
4. Overlap independent work; query embedding precedes dependent search for that query, although different queries or stages can pipeline.
5. Compare smaller models, different precision, batching and index changes on a measured cost/quality/SLO frontier rather than one mandatory order.[8] Why This order reduces avoidable work before replacing quality-bearing components. Failure without it The design can sacrifice quality while leaving context, repeated work, candidate count, and overlap opportunities untouched. Cost and complexity Context reduction, compatible-prefix caching, cascades and pipelines change resource and quality trade-offs. Measure those changes; a smaller model or cheaper index need not worsen every task, and no ordering is universally optimal.

Cost per query

What

1. Price attributable resource service and provider charges for all paths, plus fixed-capacity utilization. With rates $r_j$ per resource-hour and service $s_j$ seconds, an allocated variable proxy is $\sum_j r_js_j/3600$, not summed wall latency times one rate.
2. Add or separately report amortized ingestion, embedding, build, refresh, storage, licenses and replication under a stated time horizon and request count.
3. Evaluate retrieval-gate savings net of gate cost, missed retrieval benefits and drift; skipping is not always the cheapest or safest optimization.
4. For iterative policies report distributions of calls, tokens, cost, latency and failures, not only expected rounds.[8] Why The calculation exposes full serving cost, recurring build cost, gate value, and multi-round tail cost. Failure without it Generation tokens can hide embedding, vector-store, indexing, and repeated-retrieval costs. Cost and complexity Report per-request variable cost and allocated fixed/build/storage cost separately before a combined total. Specify utilization, currency, time horizon and traffic; latency and resource-service cost are not interchangeable.

C.5 exit criteria

What

1. Name the dominant stage and its budget share as a percentage.
2. Derive one cost-per-query number with the stages broken out.
3. For every proposed optimization, name the constraint it addresses and what it trades away. Why These gates tie optimization to measured dominance, complete cost, and explicit trade-offs. Failure without it The design can optimize a minor stage, omit costs, or hide a quality sacrifice. Cost and complexity Passing requires one percentage, one staged cost-per-query total, and one constraint-and-trade-off statement per optimization.

Diagrams

The source lists zero captioned figures and source tables. This study page includes decoder and key-number tables summarizing the five checklists, not measured experiment results.

Whiteboard pack

What to draw

1. Draw one large box labeled “Design review.”
2. Draw five cards inside it from left to right.
3. Label the cards “Retrieval,” “Indexing,” “Evaluation,” “Credibility,” and “Latency and cost.”
4. Under Retrieval, write “size, gate, sources, search, presentation.”
5. Under Indexing, write “memory, recall, churn, rebuild.”
6. Under Evaluation, write “test set, retrieval, generation, system.”
7. Under Credibility, write “sources, attribution, attacks, privacy.”
8. Under Latency and cost, write “five stages, optimize, price.”
9. Draw an arrow from every card to one bottom box labeled “Exit criteria.”
10. Inside the bottom box, write “number or named choice.”

Spoken script

Run five cards rather than merely naming them. Retrieval fixes requirements, routing, sources, scoring and presentation. Capacity separates artifacts, copies, load and operational peaks. Evaluation checks populations, evidence use and stage hypotheses without inventing guarantees. Credibility verifies permissions, source support and scoped attacks. Latency/cost traces actual dependencies and prices attributable resources. Every unknown has an owner and a validation plan; an unchecked box is not evidence of failure or success.

Interview traps

Probe 1: Why is “always retrieve” not a complete retrieval design?

Always-retrieve is an explicit gate with measured cost, not a neutral default. A complete answer also names the closed-book path, abstention behavior, controller-drift detection, sources, transformations, search depths, index target, filters, prompt placement, and attribution timing.

Probe 2: Why can vector compression fail to solve capacity planning?

Compression shrinks vector codes, not automatically identifiers, links, payloads, runtime or copy counts. Their shares depend on the exact layout; graph links are not inherently incompressible and no universal 30-fold crossover exists. Ceiling bytes provide a capacity lower bound, not a shard/replica/load plan.

Probe 3: Why is one end-to-end quality number insufficient?

It cannot localize retrieval, generation, or system failure. The evaluation checklist pairs retrieval metrics, claim-level generation measures, abstention, stage attribution, macro, micro, and mezzo views, ablations, sanity controls, latency, consistency, and corrupted-document robustness.

Probe 4: When is a cited answer still not safely grounded?

A citation can be invalid, unsupported, stale or tied to an unreliable source. Inline generation does not prove evidence use; post-hoc linking can verify support but not by itself establish the answer’s origin. Define and evaluate the precise grounding claim, enforce authorization and track source versions and removal.

Probe 5: Why not optimize search latency first?

Optimize the measured critical path and constrained resource under production load, not a presumption that search or generation must dominate. Test context, caching, cascade, overlap and component alternatives at comparable quality and risk; independently measured stage percentiles cannot simply be added.

Key numbers

00-START-HERE · 01-READING-ROUTES · A2_question_bank · A4_the_rag_card_and_index_datasheet