Appendix B: Question Bank

00-START-HERE · 01-READING-ROUTES · A1_formula_sheet · A3_design_checklists This appendix turns the book’s Retrieval-Augmented Generation (RAG) material into a timed self-test with Core, Senior, and Staff prompts plus compact answer spines.

TL;DR

• The bank contains 123 questions: one Core, one Senior and one Staff prompt for each of 41 chapters.
• Core tests mechanism and failure mode; Senior tests derivation under stated assumptions; Staff tests judgment under changed constraints.
• A numerical answer without its units, denominator, cost scope or uncertainty is incomplete.
• Explain what would falsify the diagnosis; a descriptive decomposition is not automatically causal evidence.
• Hide the answer spine immediately below each prompt, answer aloud, then follow the cited chapter for the full derivation.

The story

Imagine a flight simulator for RAG interviews. The cockpit is the interview room, the timer is the clock, and each prompt is a scenario injected by the instructor. Core drills ask whether you can identify the instrument that failed. You must name the mechanism, separate retrieval from generation, and avoid a vague diagnosis such as “the system hallucinated.” Senior drills add weather, fuel, and distance. You must calculate from the numbers in front of you. A memorized benchmark number is like yesterday’s weather report. It does not solve today’s route. Staff drills put two experienced crew members in disagreement. Both see a real risk. Your job is to price each option, concede the valid concern on each side, choose a course, and name the condition that would make you turn back. The answer key is the flight recorder. It does not fly the scenario for you. It shows the first move, the derivation to narrate, and the follow-up an interviewer is likely to press. The section pointer is the simulator manual. When the recorder gives only a short spine, the pointer leads to the full mechanism rather than inviting a guess. Run one part at a time. Set ninety seconds, speak before looking, and score whether you produced the reasoning cold. Familiarity after opening the key does not count as a safe landing. The three tiers do not rank sentence difficulty. They rank the behavior under test: mechanism, derivation, and judgment under changed constraints.

Decoder table

Core mechanics

Each of the 123 prompts below is followed immediately by a compact answer spine and its chapter pointer. Attempt the prompt aloud before revealing that spine; there is no separate B.4 answer-key section on this page. Derivations use the stated assumptions, not an automatic transfer of benchmark numbers to a new system.

B.1 Core Questions

Difficulty marker: Core. Expected of anyone who claims RAG on a resume. These prompts test whether the vocabulary is real and whether the mechanism is understood. Failing one can end a phone screen. Answer before reading the spine.

Part I - The RAG Interview Landscape

Core 1, §$1.1$. Prompt: “We fine-tuned an 8B model on our product catalog and it still invents SKUs. What do you change?” Expected reasoning: Separate memorization from a keyed source of authority. Fine-tuning can learn catalog facts, but does not provide a reliable current SKU lookup or exact row deletion. Retrieve versioned catalog records, retain an exact identifier path, apply access controls, and validate that every emitted SKU exists. Use fine-tuning for demonstrated behavior when needed; retrieval alone does not guarantee correctness.[1] Core 2, §$2.2$. Prompt: “One product fact in your assistant is wrong. You have a 27B model. Walk me through your options and pick one.” Expected reasoning: Separate weight payload from runtime. A $27\times10^9$-parameter fp32 checkpoint needs $108\,\mathrm{GB}$ for weights alone, not for an entire forward pass or training run. A local model edit can be fast in a particular experiment but needs locality and regression tests. Updating an authoritative datastore row avoids changing weights, but still requires visibility, cache and answer verification. Choose according to where the wrong fact originates.[2] Core 3, §$3.2$. Prompt: “Walk me through naive RAG and tell me what breaks.” Expected reasoning: Draw retrieval followed by generation. Diagnose missing or irrelevant evidence, failure to use sufficient evidence, and duplicated or badly bounded chunks; these are examples, not an exhaustive partition. Inspect retrieved text and source coverage before changing the generator. Record the retrieval and generation denominators separately.[3]

Part II - The Generator Side

Core 4, §$4.1$. Prompt: “Our assistant quotes last quarter’s pricing. Would you fine-tune it on the new price list?” Expected reasoning: Default to versioned retrieval for frequently changing prices, then test whether the generator follows the current authoritative record. The chapter’s training/query cost comparison is a worked FLOP proxy under specified token, hardware and throughput assumptions, not a universal price per refresh. Fine-tuning may improve behavior but does not create a dependable current-price database.[4] Core 5, §$5.1$. Prompt: “Where does the training data for a model like this actually come from?” Expected reasoning: Describe acquisition, extraction, filtering, deduplication and tokenization, naming source composition rather than calling it all “the internet.” Under the cited decimal-unit example, $20\,\mathrm{TB}\to750\,\mathrm{GB}$ retains $3.75\%$ of bytes. Tokenization need not destroy provenance: document-to-token maps can be retained. Without such records, extraction probes measure behavior and cannot enumerate historical corpus membership.[5] Core 6, §$6.1$. Prompt: “You have $10^{23}$ FLOPs of training budget. How many parameters should the model have?” Expected reasoning: State the proxy $C\approx6ND$. Under the illustrative ratio $D=20N$, $N=\sqrt{10^{23}/120}\approx2.88675\times10^{10}$ parameters and $D\approx5.77350\times10^{11}$ tokens. This is conditional training-compute arithmetic. Lifetime inference demand, quality targets, architecture and hardware can change the optimum; they do not imply a universal tenfold reduction in model size.[6] Core 7, §$7.1$. Prompt: “When would you fine-tune the generator instead of putting examples in the prompt?” Expected reasoning: Distinguish a behavior-learning objective from repeatedly supplying examples, while allowing either mechanism to carry factual information. Under the same dense-model proxy and counting all training tokens, compare $6ND_{\mathrm{SFT}}$ with $Q(2Nk t_d)$, so $Q^*=3D_{\mathrm{SFT}}/(k t_d)$. This is a compute-only break-even, not an equivalent-quality result or a fixed number of traffic days. Include attention, caching, retraining and serving overhead.[7] Core 8, §$8.1$. Prompt: “An engineer shows you an attention heat map with most of the mass on the retrieved policy chunk and concludes the model read it and overrode it. Do you accept that?” Expected reasoning: No. Attention weights indicate a possible information path, not sufficient causal evidence. The cited 2019 attention experiments motivate skepticism rather than proving that all attention is uninformative. Remove or replace the chunk under controlled decoding, compare outputs, and test redundant evidence. A zero leave-one-out effect does not prove that the chunk had no influence.[8]

Part III - Prompting and Context Construction

Core 9, §$9.1$. Prompt: “What is the difference between few-shot prompting and retrieval-augmented generation?” Expected reasoning: Both add context without updating generator weights. Demonstrations specify examples of a task; retrieved evidence supplies information for the current request. Demonstrations can also be retrieved dynamically after seeing the query, so timing is not the defining distinction. State what each context item is intended to teach or support.[9] Core 10, §$10.1$. Prompt: “Someone swapped Document 1: for [1] in your context template and grounded accuracy moved four points on a 500-question set. Real, or noise?” Expected reasoning: For one IID binary accuracy estimate, $\mathop{\mathrm{SE}}\nolimits=\sqrt{0.64(0.36)/500}\approx0.021466$, or $2.147$ percentage points; its nominal $95\%$ half-width is about $4.207$ points. That is not the standard error of a paired difference. Use the per-question discordances and a paired test or interval; four points alone does not establish either significance or insignificance.[10] Core 11, §$11.2$. Prompt: “How would you make a RAG system say ‘I don’t know’?” Expected reasoning: Compare similarity-based gates, calibrated answer scores, self-consistency and explicit insufficiency training. Similarity can reward topical rather than sufficient evidence, token probability can reward a fluent wrong answer, and repeated samples can share one error. Evaluate accepted-answer risk and coverage on representative slices, with uncertainty and an explicit abstention cost.[11]

Part IV - Representing What You Retrieve

Core 12, §$12.1$. Prompt: “Walk me through how text representation evolved, and tell me why BM25 is still running in production systems in 2026.” Expected reasoning: Explain the representational trade-offs: lexical dimensions preserve literals; static embeddings conflate senses; contextual embeddings can disambiguate but may pool away detail; multimodal spaces extend matching across inputs. These are design tendencies, not universal eras or a fixed per-language capacity tax. BM25 remains a useful exact-token baseline, especially for rare identifiers, and must be evaluated on the workload.[12] Core 13, §$13.1$. Prompt: “How do you choose a chunk size?” Expected reasoning: Choose an answer-bearing unit and measure coverage, retrieval quality and token cost. For a span of $s$ discrete tokens with uniform boundary phase in non-overlapping $L$-token chunks, severance is $\min(1,(s-1)/L)$. For stride $L-o$, $0\le o\lt L$, every span with $s\le o+1$ fits some chunk away from corpus edges; $s\le o$ is conservative. Long-run duplicated-token inflation is $L/(L-o)$, excluding edge effects.[13] Core 14, §$14.1$. Prompt: “Your RAG over PDF reports handles prose well and gets numbers wrong. Where do you look first?” Expected reasoning: Inspect whether ingestion retained the table header, row/column alignment, units and relevant cells before blaming reasoning. Retrieve a complete answer-bearing row or structured record, repeat only the needed header, and preserve the source span. For aggregations, a validated SQL path may be more appropriate. Not every table failure is an ingestion failure; test evidence sufficiency and downstream use separately.[14]

Part V - Indexing and Vector Search

Core 15, §$15.2$. Prompt: “Derive HNSW from a skip list. How many layers for ten million vectors?” Expected reasoning: For the stated random-level model, $\Pr(\ell\ge j)=M^{-j}$ and $nM^{-j}$ is the expected number reaching level $j$. With $n=10^7$, $M=16$, $\log_{16}n\approx5.813$ suggests a maximum near levels five or six, hence roughly six or seven occupied layers. Height is random, not exactly six; $\Pr(\ell_{\max}\lt j)=(1-M^{-j})^n$.[15] Core 16, §$16.1$. Prompt: “Your flat index is $30.7$ GB and you have $1\,\mathrm{GB}$ of RAM for it. Same embedding model, same dimension. What do you do?” Expected reasoning: Convert the decimal $1\,\mathrm{GB}$ budget over $10^7$ vectors into $100\,\mathrm{B/vector}$ before overhead. A 96-byte PQ code uses $960\,\mathrm{MB}$, but adding Faiss-style eight-byte IDs gives $1.04\,\mathrm{GB}$ before codebooks and runtime, so it does not fit. If equal subspaces must divide $d=768$, a 64-byte code plus eight-byte IDs uses $720\,\mathrm{MB}$; reserve remaining memory and benchmark its recall.[16] Core 17, §$17.1$. Prompt: “A user deletes a document at 10:00. Walk me through what happens in your vector store.” Expected reasoning: Distinguish read-time exclusion from physical reclamation. Publish the deletion state and enforce it on every serving path, cache and replica before claiming visibility. Verify the blocked IDs and failure behavior; “one query later” is not a universal consistency guarantee. Rebuild or compact according to the implementation, and separately handle retained logs, backups and other copies.[17]

Part VI - Retrieval and Ranking

Core 18, §$18.2$. Prompt: “What does BM25 fix relative to TF-IDF?” Expected reasoning: Contrast raw linear term counts with BM25’s saturating term-frequency factor and adjustable length normalization. Do not claim every TF-IDF system grows without bound: cosine normalization and logarithmic term-frequency variants change that behavior. Work through the chosen scorer and corpus statistics rather than treating the eleventh occurrence as universally eleven times as useful.[18] Core 19, §$19.1$. Prompt: “You have query-document pairs graded $0\text{ to }4$ by humans, and you train on them with mean squared error. Training loss halves. $\mathop{\mathrm{NDCG}}\nolimits @10$ does not move. What is wrong?” Expected reasoning: Ranking metrics depend on order, whereas MSE depends on score scale and offsets. A positive affine transformation preserves order and ties but can greatly change MSE; unchanged ranking therefore permits unchanged nDCG. State relevance gains and cutoff, and handle zero ideal DCG explicitly. This diagnoses objective mismatch, not a claim that regression can never improve rankings.[19] Core 20, §$20.1$. Prompt: “Cross-encoders are more accurate. Why not use one for retrieval?” Expected reasoning: A cross-encoder jointly scores every query-passage pair, so exhaustive scanning repeats model work for the whole corpus. Use a first-stage retriever and score only its candidates. A linear proxy is $2NT$ per pair, with attention and serving overhead added separately. The chapter’s scan time is conditional on its pair length, model and hardware; corpus count alone cannot determine a latency.[20] Core 21, §$21.1$. Prompt: “Our BM25 index misses documents that never use the query’s words. Someone proposes SPLADE. What does it actually change?” Expected reasoning: SPLADE predicts sparse vocabulary weights, so a query can assign weight to a term absent from its literal text. A BERT-style configuration may have $30{,}522$ vocabulary entries; this is not a universal SPLADE vocabulary size. Inspect learned expansion, sparsity and posting-list cost, and test whether expansion improves the intended query slices.[21] Core 22, §$22.1$. Prompt: “Your pipeline returns the right document at rank 40 and the generator never sees it. What do you change?” Expected reasoning: A gold passage at rank 40 within 100 candidates shows candidate coverage but failure to reach the final five. Reranking is a plausible intervention because it changes ordering within that pool; it cannot add missing candidates or guarantee sufficient evidence. Verify the annotation and whether the passage alone can support the answer.[22] Core 23, §$23.1$. Prompt: “In a generative retriever, where did the index go?” Expected reasoning: For an atomic identifier head with $8.8\times10^6$ documents, hidden width 768 and fp32 weights, the matrix payload is $8.8\times10^6\times768\times4=27.0336\,\mathrm{GB}$. That is one parameterization, not the cost of all generative retrieval. Sequential IDs reduce output vocabulary size but move costs into constrained decoding, training, validity and updates.[23]

Part VII - Query Understanding and Control Flow

Core 24, §$24.1$. Prompt: “A user searches for a twin bed and gets nothing useful. Walk me through the diagnosis.” Expected reasoning: Check index membership and analyzer output, term/posting counts, spelling or synonymous surface forms, and ambiguous senses. Also inspect access filters and truncation. These tests distinguish an unavailable record, a lexical mismatch and a disambiguation failure; the three named mechanisms are not an exhaustive diagnosis.[24] Core 25, §$25.1$. Prompt: “Your RAG system is less accurate than the base model on your most common queries, even though overall accuracy went up. What is happening?” Expected reasoning: Build the paired correctness table. Define $\gamma=\Pr(\text{RAG correct, closed-book wrong})$ and $\beta=\Pr(\text{RAG wrong, closed-book correct})$. Then the accuracy change is $\gamma-\beta$, regardless of the two agreement cells. An aggregate gain can coexist with a harmful slice; report counts, denominators and uncertainty.[25] Core 26, §$26.1$. Prompt: “Your assistant answers ‘Alexander Fleming’ to ‘which scientist discovered the element found in penicillin?’, and retrieval $\mathop{\mathrm{recall}}\nolimits @5$ against your gold annotations is $0.94$. What is wrong?” Expected reasoning: Separate topical relevance from evidence that answers the actual question and supports the necessary chain. A question about an unspecified “element” may be ill posed; a passage about penicillin and Alexander Fleming does not identify a chemical element. Clarify the requested relation and inspect support. High annotated recall does not prove the intended question was answered.[26]

Part VIII - Training the RAG System

Core 27, §$27.2$. Prompt: “Why would you deliberately train on examples where the answer is not in the retrieved context?” Expected reasoning: Training only on sufficient contexts leaves the missing-evidence regime poorly specified; it does not mathematically force accuracy there to zero. Measure it, distinguish closed-book correct answers from context-supported answers, and include insufficient-context examples, abstention targets and held-out verification when that matches the product requirement.[27] Core 28, §$28.1$. Prompt: “Our RAG assistant gives wrong answers. Should we fine-tune the LLM on our documents?” Expected reasoning: Diagnose retrieval coverage, evidence use and response behavior before choosing which weights to train. Retriever updates change selection; generator updates change conditional responses and can also memorize facts. Improve the stage with demonstrated headroom and include retain-set tests. Fine-tuning is not inherently limited to “behavior only.”[28] Core 29, §$29.1$. Prompt: “Why would generating queries from a document ever help retrieval? Isn’t the whole point of retrieval to match queries to documents, not the other way around?” Expected reasoning: For an indexed lexical term absent from a document, its ordinary term-matching contribution is zero. The whole BM25 document score need not be zero if other query terms match. Document-side expansion can add matching terms and changes index statistics; test those effects and do not equate an absent surface string with complete semantic absence.[29]

Part IX - Generation and Context Assembly

Core 30, §$30.2$. Prompt: “A stakeholder asks: ‘We gave the model strictly more information. How can accuracy possibly go down?’ What is the one-sentence answer, and what would you show them to make it concrete?” Expected reasoning: Use matched answer-bearing evidence placed at the beginning, middle and end, with a closed-book control and controlled token budgets. A middle-position accuracy trough is an empirical model/task result, not a theorem that long context must fall below closed-book accuracy. Test the deployed model, answer length and distractors.[30] Core 31, §$31.2$. Prompt: “Your RAG demo returns ‘It was released on September 19, 2025.’ with a citation attached. How do you decide whether that’s attributable?” Expected reasoning: Resolve the complete claim, including product, market, event and date, before checking a citation. If “September 19, 2025” appears, determine whether it denotes an announcement, availability or some other event, then verify an authoritative source. A date in a passage is not itself proof of the intended claim.[31]

Part X - Evaluation

Core 32, §$32.2$. Prompt: “Write the formula for $\mathop{\mathrm{nDCG}}\nolimits @k$ from memory.” Expected reasoning: Define gain, cutoff and the discount $1/\log_2(i+1)$; at rank one it equals one. Normalize by ideal DCG from the same judgment universe. The usual $[0,1]$ interpretation requires nonnegative gains and positive ideal DCG; state the zero-IDCG and unjudged-item policies. nDCG measures ordering under those judgments, not truth of answers.[32] Core 33, §$33.1$. Prompt: “Your team wants to add a truthfulness check to the pull-request pipeline. Someone proposes reusing the FActScore-style factuality benchmark you already run weekly. What do you check before agreeing?” Expected reasoning: Separate context-only faithfulness checks from external factuality checks. Schedule each according to oracle freshness, rate limits, cost and risk; a weekly available oracle cannot support a guaranteed daily fresh check. Use cached or bounded per-change tests, broader scheduled verification and human audits, documenting each check’s coverage and failure modes.[33] Core 34, §$34.1$. Prompt: “Your pipeline’s end-to-end accuracy dropped four points this week. What’s your first move?” Expected reasoning: Use $A=h c_1+(1-h)c_0$, where $h$ is retrieval hit rate and the two conditional accuracies use explicit evidence labels. Measure stage and slice changes on matched data, then replay frozen evidence or generation to test causes. The identity decomposes accuracy but does not by itself establish which component caused a decline.[34]

Part XI - Trust, Credibility, and Adversarial Robustness

Core 35, §$35.1$. Prompt: “Your RAG system hits a $2\%$ hallucination rate on your benchmark. Is it trustworthy?” Expected reasoning: A $2\%$ observed error rate is not a complete trust assessment. State its sample, denominator and uncertainty, then assess separate privacy, fairness, robustness, safety and operational requirements. Treat binding constraints as gates rather than allowing strong scores elsewhere to compensate for a failure. The named axes are a framework, not an exhaustive list.[35] Core 36, §$36.2$. Prompt: “What does it actually mean for a piece of content to have a valid C2PA credential?” Expected reasoning: Validate asset binding, the signature and the signer’s trusted certificate chain under the chosen policy. C2PA credentials establish scoped provenance and integrity assertions, not the truth of every statement or scene. Missing credentials do not establish fabrication. Distinguish a valid credential from an acceptable signer and an independently verified claim.[36]

Part XII - Scaling, Advanced Variants, and Design Drills

Core 37, §$37.1$. Prompt: “Your RAG endpoint’s $p_{50}$ is $1.3$ seconds and product wants it under $500\,\mathrm{ms}$. Where do you look first?” Expected reasoning: Profile retrieval, reranking, prefill, decode and queue/merge overhead on the actual request distribution. The worked chapter example is decode dominated; a measured $p_{50}$ near $1.3\,\mathrm{s}$ does not identify that bottleneck by itself. Stage medians do not generally sum to end-to-end median. Optimize the measured critical path and recheck quality and tail latency.[37] Core 38, §$38.1$. Prompt: “Why can’t you just put all the company’s document embeddings into one vector database?” Expected reasoning: Classify budget, capacity, latency and policy constraints before proposing consolidation. Hash partitions do not enforce residency or access rules, but lawful centralized or federated designs depend on the actual policy and data flows. Architecture can help enforce a rule; it cannot silently waive a binding legal restriction.[38] Core 39, §$39.2$. Prompt: “Why not just use CLIP for every image in your RAG corpus?” Expected reasoning: A single global image embedding may obscure small text, cells or layout distinctions. Patch-based late interaction is a candidate remedy, not a guarantee of accurate retrieval or reasoning. Compare page vectors, OCR/text, structured extraction and patch encodings on the layout-heavy slice, including storage, compatible reranking and evaluation of visual errors.[39] Core 40, §$40.2$. Prompt: “Why not just use an LLM to extract every triple in your corpus?” Expected reasoning: Price graph extraction and entity/relation resolution as well as storage, query retrieval and maintenance. A generated edge is not automatically true: false merges, splits and unsupported relations affect answers. Preserve evidence spans, ownership and versions, and compare graph retrieval with a simpler baseline on queries that actually require those relations.[40] Core 41, §$41.1$. Prompt: “Design retrieval for a 100 million-document corpus serving 500 QPS. What are the first two numbers you ask for?” Expected reasoning: Ask first how documents become vectors, their dimensions and precision, usable RAM, payload/metadata scope, and the task’s recall and latency targets. Document count alone is not vector count. Derive capacity and aggregate resource demand under the traffic and replication model, then benchmark rather than naming a fixed cluster from $100$ million documents and $500$ QPS.[41]

B.2 Senior Questions

Difficulty marker: Senior. These prompts test whether you can derive from the stated constraints rather than recall a favorite number. Several systems are already broken. Diagnose before proposing a fix.

Part I - The RAG Interview Landscape

Senior 1, §$1.1$. Prompt: “Our datastore just grew from 2M to 200M chunks. Can we drop the generator from 70B to 8B and bank the savings?” Expected reasoning: The corpus grows $100$-fold from two to 200 million chunks. That does not prove the prompt becomes harder, because retrieval may still deliver a small, cleaner context. Compare the 70B and 8B systems at matched evidence, coverage and prompt budgets; inspect ambiguity, contradictions and missing chains. A smaller generator is justified by measured quality and lifetime cost, not by datastore size alone.[1] Senior 2, §$2.1$. Prompt: “A regulator requires the assistant to run fully offline with no datastore, so you have to write the update into the weights. Design the run so it does not regress.” Expected reasoning: Use a controlled, localized update with retain-set replay, independent canaries and deployment rollback. None guarantees zero regression outside the test set. If replay is $5\%$ of all update tokens, its surcharge over new-data-only training is $0.05/0.95\approx5.263\%$; if it is $5\%$ of new tokens, the surcharge is $5\%$. State which denominator is intended and report uncertainty on retained performance.[2] Senior 3, §$3.1$. Prompt: “Your $p_{95}$ time to first token is $1.1$ seconds against an 800 millisecond SLO. Where do you look?” Expected reasoning: Measure the $p_{95}$ critical path and queues before changing retrieval. Query encoding, remote fan-out, reranking and prefill can all affect TTFT. Reducing prompt tokens changes linear model work and attention, while ANN scaling is implementation dependent. Test a candidate/context-depth split at matched quality and load rather than assuming any named stage explains the $1.1$-second to 800-ms gap.[3]

Part II - The Generator Side

Senior 4, §$4.1$. Prompt: “The correct passage is in the context window and the model answers from memory anyway. Which stage owns that?” Expected reasoning: If sufficient evidence is already present, inspect prompt construction, authority/version conflicts and the generator’s use of it. Improve task instructions or curated demonstrations, then consider behavior fine-tuning with retain tests if necessary. Do not treat gold presence as proof the context is unambiguous, or assume retriever and generator training are mutually exclusive.[4] Senior 5, §$5.1$. Prompt: “Legal wants written confirmation that no customer PII reached the pre-training corpus. What do you send back?” Expected reasoning: Separate a policy requirement of zero customer PII from evidence that it is met. Audit retained provenance, data handling and source records where available; tokenization does not inherently erase an external map. Without enumerability, sampled audits estimate rates and extraction probes test emissions, neither certifying zero historical exposure. Decline an unverifiable promise and specify operational controls and residual uncertainty.[5] Senior 6, §$6.1$. Prompt: “Kaplan and Chinchilla disagree by more than a factor of ten on the same question. What actually changed between them?” Expected reasoning: Explain differences in experimental scale, data, architecture, tokenizer, training durations, learning-rate schedules and fitted objectives. Chinchilla’s more balanced token/parameter allocation challenged earlier estimates within its regime. Schedule bias can matter, but it is not the only changed variable; the studies are not one identical controlled experiment. Add inference demand when optimizing a deployed system.[6] Senior 7, §$7.1$. Prompt: “Your RAG assistant scores well on the offline set and drops twelve points in production. How do you tell whether that is a retrieval failure or an ICL failure?” Expected reasoning: Localize the 12-point decline with frozen query, corpus, retriever, demonstrations and generator versions. Audit ingestion, ACLs and query construction as well as candidate recall. If evidence is absent from the fixed context, merely rewording the answer prompt cannot supply it; if demonstrations or routing changed, ICL may matter. Use controlled replay rather than choosing retrieval or ICL by intuition.[7] Senior 8, §$8.1$. Prompt: “Your leave-one-out attribution says no chunk in the context mattered, yet the answer is clearly drawn from the context. What happened?” Expected reasoning: For two interchangeable sources with value one whenever either is present, each leave-one-out marginal at the full set is zero, while symmetry gives Shapley credit $0.5$ each. The exact subset universe for 20 items has $2^{20}$ coalitions, though caching or approximation can reduce evaluations. Define the answer-value function and sampling policy; neither attribution method proves source truth.[8]

Part III - Prompting and Context Construction

Senior 9, §$9.1$. Prompt: “Your assistant returns fluent answers in the wrong format about a third of the time, and it also quotes last quarter’s prices. Where do you start?” Expected reasoning: Separate wrong format from stale facts. Structured-output constraints or demonstrations may repair the former; versioned authoritative evidence addresses the latter. Demonstrations can themselves be retrieved per query. Evaluate the two error slices independently, including whether a format improvement introduces unsupported content or suppresses a correct refusal.[9] Senior 10, §$10.1$. Prompt: “How do you compare two retrievers when the prompt format alone moves accuracy more than the retrievers differ?” Expected reasoning: A shared fixed template estimates performance under that template; independently tuned templates estimate a search procedure’s held-out performance. Use comparable development budgets, separate test data and paired uncertainty for either protocol. Report the intended estimand and interaction effects. Neither finite prompt search nor the best development score proves each arm’s population optimum.[10] Senior 11, §$11.1$. Prompt: “Your abstention rate is $54\%$ and users are leaving. Bring it down without shipping more wrong answers.” Expected reasoning: Clarify the protected objective. Lower expected harm is not the same as no increase in wrong answers. Under the stipulated perfect-checking model, checking $60\%$ of wrong answers changes cost from 20 to $0.4(20)+0.6(0.2)=8.12$. With correct reward one and abstention cost $0.5$, the utility thresholds are $19.5/21\approx0.928571$ and $7.62/9.12\approx0.835526$. Citation opens alone do not establish those checking probabilities or authorize relaxing a literal error cap.[11]

Part IV - Representing What You Retrieve

Senior 12, §$12.1$. Prompt: “Your dense retriever misses queries containing product SKUs. What do you change, and how do you prove it worked?” Expected reasoning: An unseen SKU can be tokenized into fragments; “out of vocabulary” need not mean unrepresentable. Mean pooling assigns each token vector coefficient $1/n$, but that does not prove a fixed information-loss rate. Test exact identifiers and context variants, and compare a lexical identifier path with dense retrieval. Measure rather than infer failure solely from the pooling formula.[12] Senior 13, §$13.1$. Prompt: “Semantic chunking lifted $\mathop{\mathrm{recall}}\nolimits @5$ by three points on our evaluation set. Do we ship it?” Expected reasoning: Audit synthetic boundary construction and test on real, naturally structured documents. A three-point Recall@5 gain needs sample size, paired discordances, matching corpus units and token/storage budgets before interpretation. Report boundary-crossing answer spans and ingestion failures. Synthetic seams can bias a comparison but do not automatically invalidate every observed gain.[13] Senior 14, §$14.1$. Prompt: “Design the retrieval unit for a 200-page filing containing 60 tables.” Expected reasoning: Inspect the 200-page filing’s 60 tables and classify lookup versus aggregation queries. Repeating headers can preserve row meaning; it also increases tokens according to the actual row and header lengths. The cited roughly $39\%$ inflation belongs to one worked configuration, not all tables. Preserve units and citations, and use a validated structured path where aggregation is required.[14]

Part V - Indexing and Vector Search

Senior 15, §$15.1$. Prompt: “Why not a kd-tree? It’s $O(\log N)$ and it’s in the standard library.” Expected reasoning: For Euclidean kd-tree pruning, a coordinate-plane gap greater than the search radius can exclude the opposite branch. In a high-dimensional isotropic example, typical pair distances scale like $\sqrt{2d}$ while one-coordinate gaps remain much smaller. At $d=768$, $\sqrt{1536}\approx39.192$ is a pair-distance scale, not a universal nearest-neighbor radius. Intrinsic dimension and data distribution determine actual pruning.[15] Senior 16, §$16.1$. Prompt: “Why not learn one codebook with $2^{32}$ centroids instead of four with 256? Same code length, and one codebook has to be more accurate.” Expected reasoning: Four 256-centroid subquantizers encode $4\times8=32$ bits, distinct from a 96-byte PQ configuration. A full $2^{32}\times768$ fp32 codebook uses about $13.194\,\mathrm{TB}$; four equal-subspace codebooks use $256\times768\times4=768\,\mathrm{KiB}$. The Faiss warning threshold of 39 samples per centroid would be about $1.675\times10^{11}$ samples for the full codebook versus $9{,}984$ projected training vectors per PQ subquantizer. More centroids do not guarantee better held-out recall or tractable assignment.[16] Senior 17, §$17.1$. Prompt: “How often do you re-index, and how did you pick that number?” Expected reasoning: Measure rebuild work, query arrival rate, churn-driven extra search, usable headroom and deadline constraints. Under the linear inflation scenario, average work is $g(T)=B/T+\lambda Q_0cT/2$, giving $T^*=\sqrt{2B/(\lambda Q_0c)}$ with compatible units. The unconstrained optimum is not the deployment schedule when latency, capacity or erasure obligations bind; logical exclusion and physical deletion have distinct deadlines.[17]

Part VI - Retrieval and Ranking

Senior 18, §$18.1$. Prompt: “Thirty percent of your tail queries return zero results. Where do you start?” Expected reasoning: Split the $30\%$ tail zero-results by analyzer/index membership, access eligibility, operator semantics and vocabulary mismatch. Relaxing an accidental AND can help an OR-intended query, but a pure OR zero can also reflect missing records, filtering or truncation. Use posting counts and labeled queries before replacing the matching model.[18] Senior 19, §$19.1$. Prompt: “Your labels are integers $0\text{ through }4$. Argue for and against treating them as numbers.” Expected reasoning: Grades zero through four are ordinal unless the judgment policy defines numeric utility. Choose a gain function, such as linear gain or $2^y-1$, deliberately; exponential gain is not mandatory. Score-regression loss and ranking utility still differ even when grades have meaningful spacing. Evaluate the deployed ranking metric and annotate its gain/cutoff conventions.[19] Senior 20, §$20.1$. Prompt: “Our bi-encoder hits $99\%$ accuracy on its in-batch training objective, but $\mathop{\mathrm{recall}}\nolimits @100$ on the live corpus is $61\%$. What is happening?” Expected reasoning: An in-batch $99\%$ discrimination score and live Recall@100 of $61\%$ use different candidate populations and objectives. A batch with 255 negatives is much easier than searching 21 million passages if negatives are weak or unrepresentative. Check leakage, hard negatives, false negatives and evaluation pools; compare corpus retrieval rather than treating the two percentages as inconsistent.[20] Senior 21, §$21.1$. Prompt: “We shipped SPLADE. $\mathop{\mathrm{Recall}}\nolimits @1000$ went up four points and $p_{99}$ latency went from $15\,\mathrm{ms}$ to $900\,\mathrm{ms}$. Diagnose it before you propose anything.” Expected reasoning: Going from eight to 100 active query terms is a $12.5$-fold count increase, not necessarily a $12.5$-fold posting workload. Document frequencies, pruning, cache state and index implementation determine the work. Profile posting visits and tail queries behind the 15-ms to 900-ms change; tune sparsity or expansion on a quality/latency frontier. The simultaneous recall gain does not alone establish causation.[21] Senior 22, §$22.1$. Prompt: “We shipped a cross-encoder reranker last month. End-to-end answer accuracy did not move. Was it a waste?” Expected reasoning: Define $R_2$ conditional on the first-stage hit before using $R_{\mathrm{final}}=R_1R_2$. This describes final-context evidence coverage, not answer accuracy. If candidate and final evidence recall differ by only one point on matched judgments, reranking has at most that much coverage headroom at those depths, though ordering or answer use can still improve. The first-stage ceiling may itself be poor.[22] Senior 23, §$23.1$. Prompt: “Our generative retriever returns document IDs that don’t exist in the corpus. What’s wrong and how do you fix it?” Expected reasoning: With unique fixed-width seven-digit decimal IDs, the space has $10^7$ strings. If $8.8\times10^6$ are real, $1.2\times10^6$, or $12\%$, are invalid. A trie can constrain generation to existing IDs, but valid ID output is not evidence of relevance. Prefixes, reserved IDs and collisions would change this arithmetic.[23]

Part VII - Query Understanding and Control Flow

Senior 24, §$24.1$. Prompt: “You shipped WordNet synonym expansion. $\mathop{\mathrm{Recall}}\nolimits @100$ rose 8 points and $\mathop{\mathrm{nDCG}}\nolimits @10$ fell. What happened, and what do you ship next?” Expected reasoning: Expansion can improve vocabulary coverage yet hurt top-ranked precision through drift and competition. At a fixed cutoff, adding terms need not produce a superset of results or improve recall; even Boolean OR candidates are subsequently ranked and truncated. Inspect the observed eight-point Recall@100 gain alongside nDCG@10, query slices and expansion terms. Keep only gains supported by matched evaluation.[24] Senior 25, §$25.1$. Prompt: “You have budget for exactly one gating signal. Do you take the model’s token-level confidence or a query-side popularity score?” Expected reasoning: There is no universally best single signal. Choose a pre-generation retrieval gate using held-out paired outcomes and the product’s risk/cost objective. Query popularity is timely but can proxy coverage or demographic bias; token confidence is conditional on generated text and may be miscalibrated for correctness. Compare calibrated policies, acceptance risk, saved calls and drift.[25] Senior 26, §$26.1$. Prompt: “Multi-hop accuracy is 40 points below single-hop. Do you upgrade the embedding model, raise $k$ from 5 to 50, or add a second retrieval round?” Expected reasoning: A missing bridge entity motivates iterative retrieval or decomposition, but a larger candidate pool or a better embedding can sometimes find the full chain in one round. Inspect annotated support paths and compare one-shot, deeper and iterative policies at matched latency and cost. Do not infer that increasing $k$ from five to 50 must be useless from the mechanism alone.[26]

Part VIII - Training the RAG System

Senior 27, §$27.1$. Prompt: “A policy changed last night and must be reflected by 9 a.m. An engineer proposes a nightly LoRA run over the changed documents. Adjudicate.” Expected reasoning: Under $6NT$ with $N=7\times10^9$, $T=5\times10^7$ and throughput $3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$, the run is $6{,}176.47\,\mathrm{s}\approx1.716$ GPU-hours, not $1.5$. A datastore upsert is usually more targeted for one current policy, but its embedding time depends on the encoder and token count. Neither a millisecond embedding proxy nor a completed LoRA run includes validation, visibility, cache and regression gates.[27] Senior 28, §$28.1$. Prompt: “You are on a hosted generator with no gradient access. What can you train, and what is the loss?” Expected reasoning: A text-only hosted API may expose neither weights nor token likelihoods; inspect its actual capabilities. Hard top-$k$ selection is not differentiable in the ordinary path, but soft, sampled or score-function estimators are possible with appropriate access. A frozen teacher can still supervise retrieval; explicitly define support and the direction $D_{\mathrm{KL}}(Q_{\mathrm{LM}}\Vert P_R)$ where that is the objective. Do not invent unavailable gradients or scores.[28] Senior 29, §$29.1$. Prompt: “A colleague wants to run the exact same query-generation model at query time instead. Rewrite the incoming query with an LLM rather than pre-expanding the document. Same model, same prompt style, just applied to the other argument. For a corpus of 10 million documents serving 2 million queries a day, which do you pick, and why?” Expected reasoning: With ten million document items and two million queries per day, counts cross in five days only if per-item costs are equal, there is one unchanged index version, and caching is absent. Compare actual generated/embedded tokens, churn, refreshes, storage and retrieval quality. Document expansion and query generation can have different models and failure modes, so count alone is not a cost decision.[29]

Part IX - Generation and Context Assembly

Senior 30, §$30.1$. Prompt: “Liu et al. ran both a multi-document QA task and a synthetic key-value retrieval task. Why bother with the second one if the first already shows the U?” Expected reasoning: Use synthetic key-value lookup to reduce linguistic and reasoning confounds, then a real QA task for external validity. Randomize key locations, distractors, lengths and templates while controlling model and decoding. Lookup does not prove that every confound is removed or that the same position effect will transfer to production.[30] Senior 31, §$31.1$. Prompt: “Your Bing-style assistant returns an answer with three citation marks. What exactly does each mark guarantee, and what does it not guarantee?” Expected reasoning: A citation mark is a claimed link, not proof the source was in context, causally used, supportive or true. Resolve it to a retrieved, authorized version and span; reconstruct the complete claim and verify entailment separately from independent factuality. Report citation validity, support precision and claim coverage as distinct measurements.[31]

Part X - Evaluation

Senior 32, §$32.1$. Prompt: “Two retrievers post identical $\mathop{\mathrm{precision}}\nolimits @10$ and $\mathop{\mathrm{recall}}\nolimits @10$ on your eval set. Which one do you ship?” Expected reasoning: Two top-ten result sets can tie on Precision@10 and Recall@10 but differ greatly in early ranks, so inspect MRR, nDCG and context depth. If both have Recall@5 equal to one, compare ordering, sufficiency, redundant tokens, downstream correctness and latency. Set recall alone does not prove that either is the better answer pipeline.[32] Senior 33, §$33.1$. Prompt: “Lay out what runs on every pull request, what runs nightly, and why, for a RAG product’s truthfulness evaluation.” Expected reasoning: Use bounded context-faithfulness checks on routine changes, external verification at a cadence supported by its oracle, and periodic human audits. Label stale caches, unknown evidence and verifier errors. Recall correction requires an estimable verifier-recall model rather than dividing by an arbitrary number. No fixed three-cadence plan is universally sufficient.[33] Senior 34, §$34.1$. Prompt: “Two RAG systems both score $66\%$ end-to-end accuracy on the same benchmark. Are they equally good?” Expected reasoning: The decomposition needs both conditional accuracies. At $A=0.66$, $h=0.85$, $c_1=0.75$ requires $c_0=0.15$; at $h=0.65$, $c_1=0.97$ it requires $c_0\approx0.0842857$. Both totals are 66 percent, with different diagnostic headroom. Controlled replay and improvement cost, not the identity alone, decide whether to invest in retrieval or evidence use.[34]

Part XI - Trust, Credibility, and Adversarial Robustness

Senior 35, §$35.1$. Prompt: “You have two candidate models: A is strong on five of six trust axes but weak on privacy. B is moderate but even across all six, landing at nearly the same aggregate radar area as A. Which do you ship?” Expected reasoning: Do not average away a binding privacy failure because five other axes score well. Verify each axis’s definition and uncertainty, then apply mandatory gates before a multi-objective comparison. Radar-chart area is not a policy utility and can change with axis order. Name residual risks and the conditions for release.[35] Senior 36, §$36.1$. Prompt: “Your attribution classifier reports $95\%$ precision on a benchmark, but production complaints about fabricated sources keep rising. What’s going wrong?” Expected reasoning: Check what the benchmark labeled: claim support against genuine documents does not measure source authenticity under fabricated or spoofed evidence. The reported $95\%$ precision is conditional on that evaluation distribution. Add adversarial source validation, provenance and claim-support tests with production-like base rates; do not assume every complaint identifies the classifier’s same failure mode.[36]

Part XII - Scaling, Advanced Variants, and Design Drills

Senior 37, §$37.1$. Prompt: “You cut the reranker’s candidate pool from 50 to 10 hoping to hit a latency target, and $p_{50}$ barely moved. Why, and where should you actually look?” Expected reasoning: Use the revised chapter’s planning proxy, not the sum of stage medians: 50 reranked pairs cost about $8.637635\,\mathrm{ms}$ including its attention term, versus $1{,}343.249543\,\mathrm{ms}$ total. An ideal $80\%$ reduction saves $6.910108\,\mathrm{ms}$, about $0.514\%$ of that proxy. Decode is about $90.9\%$ there. Measure actual critical paths, queues and tails before claiming a $p_{50}$ gain.[37] Senior 38, §$38.1$. Prompt: “Your team hash-sharded a single vector index across twelve shards to fix a capacity problem. Compliance flags that two shards mix EU personal data with US data. What went wrong?” Expected reasoning: Treat the stipulated residency/access rules as binding and map every shard, replica, backup, query and returned context to its permitted flows. Hash balance does not enforce jurisdiction. Repartition or federate along the allowed boundaries, then budget fan-out and failure handling. The mere presence of EU and US records is not by itself proof that every mixed store is unlawful.[38] Senior 39, §$39.1$. Prompt: “You migrate from a caption-based index to a CLIP-style shared embedding space. What changes about your reranking and thresholding, and why?” Expected reasoning: A shared embedding space does not imply equally calibrated score distributions for text, tables and images. Compare per-modality or globally calibrated policies on held-out relevance and risk targets; neither a single threshold nor separate thresholds is universally optimal. Verify encoder/projection compatibility and the reranker’s supported input types.[39] Senior 40, §$40.1$. Prompt: “Your team already runs GraphRAG over the product taxonomy. Leadership wants the same pipeline reused on a corpus of raw customer-call transcripts. What do you tell them?” Expected reasoning: Graph extraction suited to a stable, curated taxonomy may fail on informal conversations through entity ambiguity, coreference and privacy. That is a hypothesis to test, not a theorem of non-transferability. Evaluate entity/edge correctness and downstream answers, preserve evidence and permissions, and compare a lighter structured or retrieval baseline.[40] Senior 41, §$41.1$. Prompt: “Your team’s uncompressed HNSW cluster for this corpus needs ten memory-heavy nodes, and finance is asking why.” Expected reasoning: For the revised scope, raw HNSW subtotal is $3{,}200\,\mathrm{B/vector}$ versus Faiss-style 96-byte PQ codes plus eight-byte IDs, $104\,\mathrm{B/vector}$, a $30.769\times$ reduction before other costs. Five raw capacity partitions exactly fill the assumed capacity and omit positive overhead; ten replica slots are not an all-in plan. Compression may permit one partition by bytes but still needs recall, query-load and failure-domain validation.[41]

B.3 Staff Questions

Difficulty marker: Staff. These prompts test judgment after a constraint changes or two teams present partly valid positions. Price options, concede the other side’s real point, decide, and name the reversal condition.

Part I - The RAG Interview Landscape

Staff 1, §$1.1$. Prompt: “Legal needs a licensed corpus out of the product in 30 days. Research says removing it costs four points on our benchmark and refuses. Adjudicate.” Expected reasoning: First clarify the actual revocation obligation with the authorized legal owner: copies, training use, deployed weights and deadlines can differ. Do not simply move a revoked licensed corpus into a datastore and continue serving it without rights. Exclude prohibited serving copies, preserve required audit evidence, and evaluate checkpoint retirement, retained-data rebuilding or other permitted remedies. Future SILO-style separation improves reversibility but does not cure an earlier unlawful copy.[1] Staff 2, §$2.1$. Prompt: “Compliance says the reversed legal standard must be removed from the model, not merely superseded. One engineer wants to continually train on the corrected corpus. Another wants to leave the weights alone and fix it in the index. Adjudicate.” Expected reasoning: Separate making the current answer correct, suppressing the old response and removing learned influence. Retrieval plus version controls can support current answers but does not guarantee the model never states an old fact. Exact unlearning targets equivalence to a specified retained-data training procedure; finite probes cannot certify that distributional property. Choose the remedy and claim that the actual obligation requires.[2] Staff 3, §$3.1$. Prompt: “Retrieval wants to fan out across all three stores for recall. Generation wants $k=20$ for coverage. The SLO is 800 milliseconds. Both have data. Decide.” Expected reasoning: Run independent store lookups in parallel where permitted, but measure remote tail latency, encoding, merge and queue costs rather than assuming fan-out is a few milliseconds. Decouple candidate depth from generation context depth. A prefill estimate such as $769\,\mathrm{ms}$ belongs to its particular model and prompt; compare alternatives at matched quality under the 800-ms critical-path budget.[3]

Part II - The Generator Side

Staff 4, §$4.1$. Prompt: “Legal now requires any document to be removable from the system within 24 hours of a takedown notice. A colleague proposes an unlearning pass over the weights. What do you tell them?” Expected reasoning: A keyed datastore can block future retrieval only if every read path enforces deletion. It cannot remove influence already learned into weights. Approximate unlearning is a measured suppression procedure, not an exact erasure guarantee; rebuilding from correctly retained data is the chapter’s exact baseline. Apply the 24-hour requirement to the specified systems and distinguish logical exclusion, physical deletion and checkpoint retirement.[4] Staff 5, §$5.1$. Prompt: “The licence on a corpus we already fine-tuned on is revoked in thirty days. Counsel wants the checkpoint destroyed. Your ML lead says the data was $0.3\%$ of the mix and the effect is negligible. Adjudicate.” Expected reasoning: A $0.3\%$ training mixture share is not an extraction probability: duplication, prompts and model behavior matter. Equally, a probe cannot decide whether a license obligation applies. Determine provenance and the binding removal requirement, run scoped behavior tests and choose a permitted remediation with explicit uncertainty. Do not use small mixture weight as a legal or statistical exemption.[5] Staff 6, §$6.1$. Prompt: “Our research lead wants the Chinchilla-optimal 77B because it is the compute-optimal point. Infrastructure wants the 8B. We will serve $10^{10}$ tokens a day and we already run a retrieval index. Adjudicate.” Expected reasoning: At $10^{10}$ served tokens per day, a 365-day year has $3.65\times10^{12}$ tokens. Compare training plus lifetime serving work and measured quality for the 77B and 8B choices; use the actual deployment’s token mix, throughput and memory steps. A training-compute optimum need not minimize lifetime cost, and neither model is automatically preferred without the quality constraint.[6] Staff 7, §$7.1$. Prompt: “Legal has ruled that no customer record may appear in a prompt at inference time. The ML team proposes fine-tuning on those records instead. Adjudicate.” Expected reasoning: Clarify the policy’s trust boundary before changing architecture. A ban on customer records in prompts does not authorize placing those records in weights, which can memorize and expose them. Explore permitted private processing or data minimization only with authorization, and account for logs, gradients and tenant access. Do not route around a privacy requirement by relabeling the storage mechanism.[7] Staff 8, §$8.1$. Prompt: “Compliance wants a defensible per-answer attribution on every response. The research lead wants two headcount for circuit-level analysis of your 32-layer generator. You can fund one.” Expected reasoning: Price attribution under a defined value function and audit objective. Leave-one-out takes $n$ extra generations plus a baseline and can miss redundant sources; exact subset methods are costly but can be approximated. A $33^{32}$ circuit-path toy count is not a lower bound for every attribution method. With two staff, prioritize verifiable source/version/span logging and calibrated support checks, and be explicit that these do not prove a complete internal causal account.[8]

Part III - Prompting and Context Construction

Staff 9, §$9.1$. Prompt: “Compliance now requires that every claim in an answer be traceable to a source document and that the model’s own knowledge is never used. A senior engineer proposes adding ‘answer only from the provided context’ to the system prompt and shipping. Adjudicate.” Expected reasoning: Translate “no model-only knowledge” into a testable answer-support policy. The generator necessarily uses parameters, so citations cannot establish zero parametric influence. Require each factual claim to have an accessible, authorized supporting span, verify coverage and support, and abstain or escalate when evidence is insufficient. A prompt instruction alone does not enforce that contract.[9] Staff 10, §$10.1$. Prompt: “Your team spent two weeks tuning the context template for a 7-point gain. Platform now wants to move the product to a hosted frontier model, and a principal engineer argues the tuned template is the team’s asset and must be carried over verbatim. Adjudicate.” Expected reasoning: A prompt can transfer across models, but a seven-point gain on one model does not establish a gain on another. Preserve the evaluation cases, error taxonomy and tuning protocol; revalidate the template on separate development and held-out data at matched budget. Avoid assuming either universal transfer or universal invalidity of the old string.[10] Staff 11, §$11.1$. Prompt: “The clinical safety lead demands $99\%$ correctness before the system answers anything. The head of product says a policy that refuses nine questions in ten is not a product. Adjudicate.” Expected reasoning: Define correctness among accepted answers, its confidence interval, minimum coverage and the clinical consequence of error. An $80\%$ macro-$F_1$ score does not imply a $94.5\%$ precision ceiling; selective precision depends on score distributions and operating points. Evaluate risk/coverage curves and representative clinical slices. If no validated operating point meets both requirements, escalate that measured conflict rather than inventing a mathematical impossibility.[11]

Part IV - Representing What You Retrieve

Staff 12, §$12.1$. Prompt: “Platform wants to consolidate five per-language indexes onto one multilingual model and claims a five-fold cost saving. Search says accuracy will drop. Adjudicate.” Expected reasoning: Consolidation can save deployments and operational duplication, but it saves vector bytes only when items can be deduplicated or represented more compactly at the same required quality. Account for model/dimension/precision differences and migration copies. Compare shared infrastructure with jurisdiction, tenant and failure-isolation constraints; five indexes do not imply either zero savings or five times the necessary staffing.[12] Staff 13, §$13.1$. Prompt: “Legal requires every retrieved chunk to carry a clause citation. Platform requires uniform 512-token chunks so ANN latency stays predictable. Adjudicate.” Expected reasoning: Changing to 512-token chunks affects vector count, evidence integrity and downstream prompt length. In a one-vector-per-chunk encoder it does not directly change vector width, but that assumption excludes late-interaction or variable-vector encodings. Measure both index geometry and generation cost before choosing a chunk size to fit a prefill budget.[13] Staff 14, §$14.1$. Prompt: “The data team wants the tables extracted into a warehouse and queried with text-to-SQL. The search team wants everything in one vector index. Adjudicate.” Expected reasoning: Classify point lookups versus joins, arithmetic and aggregation. Header-repeated rows can support lookup; a validated schema/SQL path may support aggregates more reliably. The worked roughly 420-token row is not a universal size. Enforce authorization and provenance on either path and compare table-level, row-level and structured retrieval on real questions.[14]

Part V - Indexing and Vector Search

Staff 15, §$15.1$. Prompt: “A colleague cites Beyer et al. and argues that nearest neighbor is meaningless at 768 dimensions, so semantic search over our corpus is theoretically doomed. Our offline eval says $\mathop{\mathrm{recall}}\nolimits @10$ is $0.82$. Adjudicate.” Expected reasoning: Distance-concentration results depend on distribution, norm and dimensional regime, not merely the number 768. Real learned data can have lower intrinsic dimension or clusters, but contrastive training does not guarantee that theorem assumptions fail. Clarify whether Recall@10 of $0.82$ measures ANN recovery of exact neighbors or task relevance, and benchmark both before deciding what the theorem implies.[15] Staff 16, §$16.1$. Prompt: “A colleague wants binary quantization instead. Also 96 bytes at $d=768$, also $2^{768}$ possible codes, no codebook, no training, and Hamming distance by popcount. Adjudicate.” Expected reasoning: Both 96-byte codes have at most $2^{768}$ bit patterns, but cardinality does not determine distortion or retrieval quality. Unrotated sign quantization fixes coordinate orthants; learned or rotated binary methods also exist. Ninety-six eight-bit PQ subquantizers learn separate centroid choices. Compare encoding, query scoring, memory and recall at matched data and training budget; neither representation wins from bit count alone.[16] Staff 17, §$17.1$. Prompt: “Legal now requires erasure within 24 hours, and your rebuild takes an hour on a corpus growing to 500 million. Platform wants to keep the nightly job. Adjudicate.” Expected reasoning: Find the baseline corpus and measure scale. If one hour at ten million items scaled exactly linearly to 500 million, it would be 50 hours; the revised chapter’s $1.092267$-hour proxy instead gives $54.6133$ hours. Neither is a measured scaling law. A nightly full rebuild is not an erasure guarantee: enforce verified read-time exclusions promptly, then meet the separate physical-copy and backup obligations on their required schedules.[17]

Part VI - Retrieval and Ranking

Staff 18, §$18.1$. Prompt: “Legal requires that answers cite only documents the user is cleared to see. The search team wants every filter moved into the scoring function, because hard filters over-restrict and are costing them recall. Adjudicate.” Expected reasoning: Keep mandatory security clearance as a hard eligibility constraint. Evaluate recall within the authorized corpus, not against documents the user must never receive. Improve metadata quality, fail-closed behavior, query interpretation and ranking among eligible items; a preference-filter relaxation is not permission to relax access controls.[18] Staff 19, §$19.1$. Prompt: “A colleague points out that your new labels are not human grades at all. They are dwell time in seconds, a genuine continuous quantity. They want to switch the loss back to regression. Do you agree?” Expected reasoning: Continuous dwell-time labels can make a numeric regression scale meaningful, but dwell is also confounded by rank, exposure and user intent. Regression can be useful; it is not automatically aligned with the product’s ranking utility. Model or test those biases, choose a loss and metric deliberately, and validate downstream task outcomes rather than dismissing or accepting MSE categorically.[19] Staff 20, §$20.1$. Prompt: “The retrieval team wants to ship a new embedding model monthly. The platform team says the corpus is 500M chunks and refuses. Adjudicate.” Expected reasoning: Document-encoder changes usually invalidate cached document vectors unless a tested compatibility mechanism exists. Query-only updates need not force a full passage re-embedding. Price monthly refreshes of 500 million chunks, version compatibility, shadow indexes and rollback; compare frozen document encoders, distillation or adapters against joint-training gains at matched total lifecycle cost.[20] Staff 21, §$21.1$. Prompt: “The platform team will fund exactly one index for a 500M-chunk corpus. The query log is half natural language and half part numbers and error codes. The ML team wants dense. The search team wants BM25. Adjudicate.” Expected reasoning: The dense fp32 vector payload alone is $5\times10^8\times768\times4=1.536\,\mathrm{TB}$, excluding index/runtime costs. That does not prove dense retrieval loses every literal query. Under the one-index constraint, benchmark BM25, dense and a feasible learned-sparse or single-engine hybrid alternative on the mixed log, preserving exact-identifier tests. Recommend the best measured feasible design, not an unfunded second index.[21] Staff 22, §$22.1$. Prompt: “Context windows are a million tokens now. The platform team wants to delete the reranker and pass the top 200 chunks straight through. The search team refuses. Adjudicate.” Expected reasoning: Two hundred 256-token chunks add $51{,}200$ tokens before query, instructions and output headroom. Fitting a nominal million-token window does not establish latency, quality or cost. Include linear model work, quadratic attention where applicable, caching and reranking. Compare a reranked smaller context at matched evidence coverage rather than assuming reranking is always negligible or long context is always worse.[22] Staff 23, §$23.1$. Prompt: “Infra says moving to generative retrieval deletes the vector-store line item. Retrieval says it just relocates the index onto the GPU and the bill goes up. Who is right, and what happens when the corpus grows 10 times?” Expected reasoning: Generative retrieval changes where index state lives rather than deleting maintenance and serving cost. A small sequential-ID model is not the same parameterization as an atomic document-output matrix. Compare actual model weights, tries, retained text, training, update time and measured retrieval quality; published 29.3-GB/440-MB figures are not a universal like-for-like compression factor. Tenfold corpus growth needs a fresh scaling and validity test.[23]

Part VII - Query Understanding and Control Flow

Staff 24, §$24.1$. Prompt: “Your search lead wants to retire BM25 for a dense bi-encoder to end vocabulary mismatch. On-call objects that head queries will regress. Thirty percent of traffic is exact part numbers. Adjudicate.” Expected reasoning: Retain exact-identifier capability for the stipulated $30\%$ of traffic, whether through lexical matching, routing or a tested compatible single system. Dense matching can help polysemy when context disambiguates it, but cannot resolve genuinely underspecified queries by geometry alone. Compare slice quality and whole-system cost before replacing the current path.[24] Staff 25, §$25.1$. Prompt: “Infrastructure wants the gate to cut retrieval calls by half. Quality will not accept any accuracy regression. Adjudicate.” Expected reasoning: Halving retrieval calls with no accuracy regression is a measurable policy goal, not inherently impossible. Use the paired RAG/closed-book contingency table and estimate which skipped queries gain or lose. A gate could safely skip many zero-benefit calls, but imperfect prediction and drift may create a trade-off. Validate savings, confidence bounds and protected slices; do not infer incompatibility from one threshold or the aggregate skip rate.[25] Staff 26, §$26.1$. Prompt: “Retrieval owns $\mathop{\mathrm{recall}}\nolimits @5$ of $0.94$ and says the generator is broken. Generation says the context is wrong. Your $p_{95}$ budget is $800\,\mathrm{ms}$ and you cannot add a round to every query. Adjudicate.” Expected reasoning: Recall is defined by its evidence judgments and can include complete chains and freshness. High topical Recall@5 alone does not clear either retrieval or generation. Annotate required support and version validity, replay matched contexts, and compare one-shot versus iterative policies within the 800-ms $p_{95}$ constraint. Do not assign blame or reject both teams solely from the aggregate metric.[26]

Part VIII - Training the RAG System

Staff 27, §$27.1$. Prompt: “Retrieval and generator teams each want the same quarter of GPU budget. You have $A_{\mathrm{orc}}=0.82$, $A_{\mathrm{ng}}=0.31$, $R=0.75$. Decide, and say what would reverse the decision.” Expected reasoning: Under the stated approximation, $A\approx A_{\mathrm{ng}}+R(A_{\mathrm{orc}}-A_{\mathrm{ng}})$ gives $\partial A/\partial R=0.51$, $\partial A/\partial A_{\mathrm{orc}}=0.75$ and $\partial A/\partial A_{\mathrm{ng}}=0.25$. These are slopes per metric unit, not per GPU-hour or dollar. Estimate attainable gains, training/refresh cost and interactions; oracle-context gains may not transport to retrieved contexts. The three slopes alone cannot allocate the budget.[27] Staff 28, §$28.1$. Prompt: “Research wants joint end-to-end training. Platform points out the corpus turns over $5\%$ per week. Adjudicate.” Expected reasoning: If $5\%$ of a ten-million-chunk corpus changes, refresh $500{,}000$ chunks with the frozen document encoder, then validate and publish versions. The about-83-second proxy assumes 256 tokens per chunk, a 110M encoder, $2NT$ work and $3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$, excluding I/O and other overhead. Changing the document encoder can require a full refresh; query-only or compatible updates need not. Compare joint-training benefits with that lifecycle cost.[28] Staff 29, §$29.1$. Prompt: “Retrieval metrics improved after you shipped document-side query expansion, but a stakeholder insists users still can’t find internal documents they know exist, and blames ‘the model doesn’t know it.’ Two engineers disagree. One wants to regenerate predicted queries with a larger model. The other wants to leave retrieval alone and change the generation prompt. How do you adjudicate?” Expected reasoning: Localize the failure: absent from the index, unauthorized, not retrieved, outside the candidate cutoff, or present but unused. Query generation can sometimes improve candidate coverage through vocabulary changes; a different answer prompt cannot add evidence to an already fixed context. Choose the intervention that can affect the measured failure, rather than ruling out both proposals before inspecting candidates.[29]

Part IX - Generation and Context Assembly

Staff 30, §$30.1$. Prompt: “Your team wants to raise top-$k$ retrieval from 5 to 20 because $\mathop{\mathrm{recall}}\nolimits @20$ is far higher than $\mathop{\mathrm{recall}}\nolimits @5$. A colleague objects that this will hurt end-to-end accuracy. Who is right, and what would settle it?” Expected reasoning: For one fixed ranking, Recall@20 cannot be lower than Recall@5, but answer quality need not increase when the extra items enter the prompt. More distractors, changed positions and token costs can hurt. Test content/position-controlled contexts and reranking, then route depth according to evidence requirements, quality and latency distributions.[30] Staff 31, §$31.1$. Prompt: “Your generation team wants to strip citation markers from answers because a user study found citations don’t move satisfaction scores. Product wants to keep them for a compliance audit next quarter. How do you adjudicate?” Expected reasoning: Distinguish user-facing citation UX from auditability and support verification. Hiding citations may improve one satisfaction measure without satisfying a compliance requirement. Preserve source versions and claim/span evidence where authorized even if UI markers change, and measure support coverage separately. A citation is neither a truth guarantee nor sufficient causal attribution.[31]

Part X - Evaluation

Staff 32, §$32.1$. Prompt: “Your reranking team says retrieval-stage metrics don’t matter, because the reranker fixes the ordering anyway. Your generation team says garbage in, garbage out. Adjudicate.” Expected reasoning: A pure reranker can only reorder the handed-off candidates. Measure candidate recall and final support coverage before changing it; add retrieval or decomposition if required evidence lies outside that set. A high first-stage score does not prove all multi-hop support is available, and a reranker with an actual retrieval action must be described as a different pipeline.[32] Staff 33, §$33.1$. Prompt: “Right after a vector-store migration, your nightly factuality pipeline starts flagging a much higher share of your RAG system’s answers as false. The eval team says the RAG system regressed. Infra says the eval’s external verifier broke during the migration. You have two minutes before the standup. Adjudicate.” Expected reasoning: Run a fast controlled replay with frozen prompts, contexts and versions, while independently checking verifier health and sample uncertainty. Flat measured faithfulness with lower factuality can reflect changed evidence, a generation shift, label distribution or a broken external oracle; it does not uniquely locate the fault. A two-minute triage is a scoped diagnostic, not a final causal or release-safety guarantee.[33] Staff 34, §$34.1$. Prompt: “Your retrieval team says $\mathop{\mathrm{recall}}\nolimits @k$ is $92\%$, clean. Your generation team says faithfulness, measured against gold context in isolation, is $95\%$, also clean. End-to-end correctness is $58\%$. Both teams blame each other. Adjudicate.” Expected reasoning: Recall $92\%$, faithfulness $95\%$ and end-to-end correctness $58\%$ may use different questions, units and denominators; do not multiply them as an accuracy guarantee. Audit full support chains, freshness, instruction compliance and the deployed prompt distribution. Replay the same items with oracle evidence and controlled generation to locate usable headroom.[34]

Part XI - Trust, Credibility, and Adversarial Robustness

Staff 35, §$35.1$. Prompt: “Compliance now bans any personal data from entering the retrieval context. Two teams disagree on how to update the trust dashboard. One wants to zero out the privacy axis for every model going forward. The other wants to remove the axis from the chart entirely since it no longer applies. Which do you pick, and why?” Expected reasoning: A privacy restriction remains a deployment obligation even if ingestion is designed to exclude personal data. Verify that exclusion and inspect queries, logs, backups, embeddings and model exposure. Do not set the privacy score to zero or remove the axis without explanation. Record applicable controls, evidence, uncertainty and residual risk; score conventions must not silently change the release gate.[35] Staff 36, §$36.1$. Prompt: “Legal wants every retrieved document to carry a verified content credential before it can be cited. Product says most of the corpus predates credentialing, and a hard gate would gut recall. How do you resolve this?” Expected reasoning: Credentials are scoped evidence, not truth probability one; absent credentials are not truth probability zero. A graded provenance policy may preserve useful legacy sources only when it satisfies mandatory rules. If credentials are a hard legal gate, do not score around it: seek an authorized exception, another source or escalation. Evaluate authenticity, claim support and truth separately.[36]

Part XII - Scaling, Advanced Variants, and Design Drills

Staff 37, §$37.1$. Prompt: “Product wants a $500\,\mathrm{ms}$ $p_{50}$. Engineering wants to swap the 8B generator for a 30B model to fix a quality regression, keeping the same 2,048-token context and 150-token answers. Adjudicate.” Expected reasoning: The revised planning example totals about $1{,}343.25\,\mathrm{ms}$, about $2.6865$ times a 500-ms target; this proxy is not a measured median. Moving 8B to 30B multiplies equal-precision weight payload by $3.75$, not necessarily all request time. Measure decoding, batching, quantization, caching and quality under the fixed context/output workload. Do not promise that a larger generator meets the target without a demonstrated serving plan.[37] Staff 38, §$38.1$. Prompt: “A director wants to merge four departmental RAG systems into one to cut infrastructure cost by a projected $40\%$. Legal says two of the four departments hold data that cannot legally be merged. Adjudicate, and tell the director what you’d actually ship.” Expected reasoning: Treat the stipulated jurisdictional separation as binding while evaluating the projected $40\%$ saving. A federated query surface can still transfer private queries, retrieved text, embeddings or logs across boundaries, so map those flows and obtain the authorized policy interpretation. Consolidate only permitted components and compare real duplicated overhead, tail latency and failure isolation. The projected saving is not established merely by merging four deployments.[38] Staff 39, §$39.1$. Prompt: “Your team is split. One engineer wants to migrate the entire corpus to a ColPali-style document-as-image index because it eliminates OCR error. Another says an 86-fold storage increase is not worth it for a corpus that is $95\%$ plain prose. Infrastructure budget is fixed. Adjudicate.” Expected reasoning: Treat 86-fold storage as a layout-specific estimate, not a universal ColPali factor or error guarantee. Under equal per-item baseline payloads, applying that factor only to a $5\%$ slice gives $0.95+0.05(86)=5.25$ times baseline payload, before index/serving differences. Compare that selective plan with OCR/text and structured extraction on real layout queries; page images avoid OCR transcription but still incur visual retrieval and reasoning errors.[39] Staff 40, §$40.1$. Prompt: “Infra wants to freeze the graph index because a full re-extraction costs $\$2{,}850$ and six hours. Product wants same-day updates for a compliance SLA. Adjudicate.” Expected reasoning: The quoted $\$2{,}850$ and six hours are scenario inputs for a full extraction run, not universal GraphRAG cost. Incremental entity updates can help, but entity resolution, connected edges and community summaries may propagate changes beyond touched rows. Block expired material at read time, version the graph and publish validated updates atomically. Demonstrate that extraction, propagation and cache handling meet the same-day SLA.[40] Staff 41, §$41.1$. Prompt: “Product wants QPS to jump from 500 to $5{,}000$ for a new real-time surface. The corpus stays at 100 million documents. One engineer wants to add shards. Another wants to add cores to the existing shards. Adjudicate.” Expected reasoning: A tenfold QPS rise at fixed corpus size is a load change, not a new byte-capacity requirement. At $8\,\mathrm{ms}$ of CPU service per query and $50\%$ target utilization, 500 and $5{,}000$ QPS require about eight and 80 aggregate cores per logical shard under that model. Replicas or more cores are options; extra logical shards still see fan-out but may reduce per-query work. Benchmark both and include queues, failure capacity and routing.[41]

Diagrams

The source appendix lists zero captioned figures and zero source tables. This study page adds a decoder table and a key-number table; they summarize the drills rather than represent empirical results.

Whiteboard pack

What to draw

1. Draw one large box labeled “123-question simulator.”
2. Split it into three lanes labeled Core, Senior, and Staff.
3. Put “41 questions” under each lane.
4. Under Core, draw “name mechanism” leading to “name failure.”
5. Under Senior, draw “diagnose” leading to “derive from constraints” and then “state number.”
6. Under Staff, draw two partly correct teams converging on “price options,” “concede,” “decide,” and “reversal condition.”
7. Draw a ninety-second timer before all three lanes.
8. Draw a closed answer key after the timer, then an arrow from the key to the source-section pointer.

Spoken script

Treat this appendix as a three-level interview simulator. Core questions test whether I can name the mechanism and its failure mode. Senior questions make me derive from the stated constraints, so I diagnose first and put a number in the answer. Staff questions introduce a changed constraint or two partly correct teams. I price both options, concede each valid concern, choose, and state what would reverse me. There are 41 questions per tier, 123 total. I set a ninety-second timer, answer aloud before opening the key, then use the section pointer when the compact reasoning spine is not enough.

Interview traps

1. Why separate the prompts from their answer spines?

Hide or cover the answer spine directly beneath the prompt before attempting it. Seeing an answer tests recognition rather than cold production; the ninety-second limit is a practice target, not a validated hiring threshold.

2. Are Core, Senior, and Staff labels just easy, medium, and hard?

No. Core tests mechanism, Senior tests derivation from given constraints, and Staff tests judgment when constraints change or credible teams disagree.

3. What makes a Staff answer stronger than choosing the team you agree with?

A strong answer prices both options, concedes the other side’s valid point, and states a recommendation. It also names the condition that would reverse that recommendation.

4. Why should a Senior answer derive a number instead of quote one from memory?

The prompt’s constraints define the current system. A remembered benchmark number can describe a different corpus, load, or budget and therefore avoid the actual derivation.

5. What should you do when the compact answer key feels obvious or incomplete?

First ask whether you could have produced that spine cold and aloud. Then follow the attached section pointer, which leads to the full argument and expected follow-up.

Key numbers

00-START-HERE · 01-READING-ROUTES · A1_formula_sheet · A3_design_checklists