Appendix B Common Interview Questions

Each entry below gives the question as an interviewer would phrase it, a 3-6 sentence answer skeleton (the spine, not the full argument), a pointer to the section that develops the argument in full, and a senior follow-up with a short sketch of what separates a strong answer. Read the referenced sections first; these sketches are revision aids, not substitutes.

B.1 Architecture and Training Questions

1. Walk me through a decoder-only transformer block. What are the key sub-components, and in what order does data flow? Skeleton. Each block contains a self-attention sub-layer and an MLP sub-layer, each wrapped in a pre-norm residual. The input is added back to the output of each sub-layer (residual stream). Pre-norm placement - normalizing before the sub-layer rather than after - keeps the residual signal intact across depth and is the near-universal choice in models trained past 1B parameters. The MLP is typically SwiGLU with $d_{f f} \approx \lfloor 8 d / 3 \rfloor$, making it roughly $2 \times$ the parameter budget of the attention sub-layer per layer. See Section 3.1. Senior follow-up. “Why is the MLP parameter count so sensitive to the $d_{f f}$ formula, and how does it shift when you move from ReLU to SwiGLU?” A senior answer notes that SwiGLU requires three weight matrices ($W_{1} , W_{2} , W_{3}$) compared to two for ReLU MLP, so the factor of $\lfloor 8 d / 3 \rfloor$ rather than $4 d$ is a deliberate FLOPs-matched downscale to keep total parameter count comparable; naive adoption of SwiGLU without adjusting $d_{f f}$ inflates MLP cost by $\sim$50%.

2. Why do modern LLMs use RMSNorm instead of LayerNorm? Isn’t the FLOP difference negligible? Skeleton. The FLOP difference is indeed negligible (normalization is $<$0.2% of total FLOPs), but the argument is about memory bandwidth, not arithmetic. Normalization kernels are memory-bandwidth-bound; on an H100, peak compute/bandwidth ratio is $\sim 300 \times$, meaning any kernel doing fewer than 300 FLOPs per byte hits the memory wall. LayerNorm’s extra $\beta$ parameter adds a full parameter-load pass over $d$ scalars per call; across 64 normalization calls in a 32-layer 7B model, that is $\sim$524 KB of avoidable HBM traffic per sample. Ablations show quality parity and $\sim$5% throughput improvement. See Section 3.2. Senior follow-up. “Walk me through the roofline model for a LayerNorm kernel. What is its arithmetic intensity, and how does that determine the speedup ceiling from switching to RMSNorm?” A senior answer computes arithmetic intensity ($\sim$3-7 scalar ops per element loaded), places it firmly left of the roofline ridge, and concludes that any reduction in memory traffic reduces wall time proportionally - giving $\sim$12.5% reduction in normalization time, which maps to 3-5% overall throughput gain.

3. What activation function do frontier models use in the MLP, and why? Skeleton. The dominant choice is SwiGLU: $\text{SwiGLU} ( x , W , V , W_{2} ) = ( x W \odot \sigma ( x V ) ) W_{2}$, a gated variant of SiLU. The gate $\sigma ( x V )$ acts as a soft input filter that empirically yields better loss at matched FLOP budgets than GeLU or ReLU, and it enables smooth gradient flow without dead neurons. The cost is the third matrix $W_{3}$; production stacks adjust $d_{f f}$ downward to compensate. See Section 3.3. Senior follow-up. “If SwiGLU needs three weight matrices and GeLU only two, how do you set $d_{f f}$ so the FLOP budget stays the same as a GeLU baseline with $d_{f f} = 4 d$?” The answer sets $d_{f f}^{\text{SwiGLU}} = \lfloor 8 d / 3 \rfloor$,

restoring parameter and FLOP parity; a senior candidate derives this from the $3 \times d \times d_{f f}$ vs. $2 \times d \times 4 d$ parameter comparison.

4. Explain RoPE. How does it differ from absolute positional embeddings, and why is it preferred for long-context models? Skeleton. Rotary Position Embedding encodes position by rotating query and key vectors in the complex plane: $q_{m} \rightarrow q_{m} e^{i m \theta}$, so the attention dot product between position $m$ and $n$ depends only on $m - n$ - a relative offset. Absolute embeddings inject position into the residual stream before any layers, which makes extrapolation to unseen lengths fragile. RoPE is applied at every attention head and generalizes better under length extension techniques (e.g., YaRN, RoPE scaling). See Section 3.4. Senior follow-up. “What goes wrong when you try to extend a RoPE model to context lengths it was not trained on, and how does YaRN address it?” High-frequency RoPE dimensions wrap around at lengths beyond training distribution, causing attention scores to become noisy. YaRN interpolates rotation frequencies so that no dimension exceeds its trained range; a senior answer explains the interpolation factor $s = L^{\prime} / L$ and why the NTK-aware variant is preferred over naive linear interpolation.

5. You need to design the attention hyperparameters for a 70B model: how many heads, how many KV heads, and what head dimension do you choose? Skeleton. Head dimension is typically fixed at 128 (matching one warp on modern GPUs and fitting Flash Attention block sizes efficiently). At $d = 8192$ for a 70B model, that gives 64 attention heads. KV heads are reduced via Grouped-Query Attention (GQA) to 8 or fewer, so each KV head is shared by 8 query heads; this cuts the KV cache by $8 \times$ with negligible quality loss. Multi-Head Attention at this scale is now unusual - MQA or GQA is the default for any model intended for serving. See Section 4.2. Senior follow-up. “How does the number of KV heads interact with tensor parallelism degree? What breaks if you tensor-parallel over 8 GPUs but only have 4 KV heads?” Each GPU in tensor-parallel must hold at least one KV head; if the TP degree exceeds the KV head count, KV heads must be replicated, partially negating the cache savings. A senior candidate designs the KV head count as a multiple of the largest anticipated TP degree.

6. What causes loss spikes during large-scale training, and what mechanisms do modern stacks use to prevent or recover from them? Skeleton. Loss spikes are typically triggered by outlier gradient norms driven by softmax instability in the attention layers: logit values grow large, the softmax saturates, and gradient flow collapses. Mitigation strategies include global gradient norm clipping (usually at 1.0), QK-norm (normalizing queries and keys before the dot product to bound logit growth), and z-loss (a regularization term penalizing large logit magnitudes applied to the final output softmax). Soft logit capping (Gemma, Gemini) bounds logits to a fixed range without hard clipping. See Section 5.1. Senior follow-up. “If you observe a spike at step 12,000 in a 100B run, how do you decide whether to roll back, adjust a hyperparameter, or continue?” A senior answer distinguishes recoverable spikes (gradient norm returns to baseline within 50-100 steps, loss follows) from unrecoverable divergence (norm stays elevated, loss plateaus above pre-spike level). The decision tree: roll back to the last checkpoint if loss does not recover within 200 steps; if the spike is recoverable, log the step and batch index for contamination investigation before resuming.

7. Why does MoE offer better loss-per-FLOP than a dense model of the same active-parameter count? Skeleton. A dense model uses the same parameter set for every token; an MoE routes each token to the $K$ highest-scoring experts out of $E$ total, so the effective parameter count seen per token is $K / E$ of total

MoE parameters - much less than the full model - while the total unique parameters encode a far richer function space. At matched active-FLOPs budgets (same $K \times d_{f f}$), the MoE has $E / K$ more unique parameters, which translates directly to lower loss at the same training compute. See Section 6.1. Senior follow-up. “What are the three main failure modes of token-choice top-K routing, and how does auxiliary-loss-free load balancing address them?” The three failure modes are load collapse (all tokens route to one expert), token dropping (overflow when an expert exceeds its capacity), and routing instability (oscillation during early training). Auxiliary-loss-free balancing tracks expert load via a running EMA bias term and adjusts routing scores to steer load toward underloaded experts without adding a gradientcompeting auxiliary loss term. See Section 7.2.

8. How do you choose between data parallelism, tensor parallelism, and pipeline parallelism for a 100B training run? Skeleton. Data parallelism (ZeRO-3) shards optimizer state, gradients, and parameters across replicas with no inter-GPU communication inside the forward pass; it is the default until a model’s parameter count exceeds per-GPU memory. Tensor parallelism partitions individual weight matrices across devices, requiring two all-reduce calls per layer - it should be limited to GPUs on the same NVLink domain (typically $\leq 8$) where bandwidth is high. Pipeline parallelism partitions layers across nodes and requires careful micro-batching to hide bubble overhead; ideal bubble fraction is $\approx 1 / m$ where $m$ is the number of micro-batches per pipeline flush. A 100B run on 512 GPUs typically uses TP$= 8$, PP$= 4$-$8$, DP$=$remaining. See Section 12.4. Senior follow-up. “At what point does tensor parallelism overhead exceed its memory benefit, and how do you detect this empirically?” Profile MFU (Model FLOP Utilization) as TP degree increases; if MFU drops more than $\sim$5% relative per doubling of TP, communication overhead is dominating. This typically occurs when TP crosses NVLink boundaries (cross-node all-reduce at $\sim$200 GB/s vs. intra-node $\sim$900 GB/s); at that point, switching to additional pipeline or data parallelism is preferred.

B.2 Inference and Serving Questions

1. Explain the KV cache. Why does generation have fundamentally different GPU utilization characteristics than prefill? Skeleton. During prefill, all prompt tokens are processed in parallel: the GPU receives a large matrix multiply (high arithmetic intensity, compute-rooflined). During generation, one new token is produced per step and attends to all cached KV pairs: this is a matrix-vector multiply (low arithmetic intensity, memory-bandwidth-rooflined). The KV cache stores the $K$ and $V$ projections for every prior token so they are not recomputed; its size grows as $2 \times L \times d_{h e a d} \times n_{h e a d s} \times T$ bytes per sequence. See Section 15.1. Senior follow-up. “How does KV cache size constrain maximum batch size at serving time, and how does paged attention address this?” KV cache for $B$ sequences of length $T$ occupies $B \times 2 L T d$ bytes; at large $T$, this exhausts VRAM before the GPU’s compute is saturated. Paged attention stores KV blocks in non-contiguous VRAM pages (analogous to virtual memory), enabling fine-grained memory sharing (prefix caching) and eliminating fragmentation from variable-length sequences.

2. What is GQA and how does it reduce KV cache pressure? Skeleton. Grouped-Query Attention assigns multiple query heads to share a single KV head: if there are $H_{Q}$ query heads and $H_{K V}$ KV heads with $H_{K V} < H_{Q}$, each KV head is replicated to serve $H_{Q} / H_{K V}$ queries. The KV cache footprint scales with $H_{K V}$ , not $H_{Q}$, so setting $H_{K V} = H_{Q} / 8$ reduces cache size

by $8 \times$ with minimal quality loss. Multi-Query Attention (MQA) is the limiting case $H_{K V} = 1$; GQA is the practical middle ground used by LLaMA 2/3, Mistral, and Gemma. See Section 16.1. Senior follow-up. “At inference time, how does the GQA KV head count interact with tensor parallelism? What breaks if $H_{K V}$ is not divisible by the TP degree?” Each TP shard must hold complete KV heads; if $H_{K V}$ is not divisible by TP, KV heads must be replicated across shards, negating cache savings. A senior design ensures $H_{K V}$ is a multiple of the maximum anticipated TP degree (e.g., $H_{K V} = 8$ for TP up to 8).

3. Explain speculative decoding. When does it help and when does it not? Skeleton. A small draft model generates $K$ token proposals in one pass; the large target model verifies all $K$ in parallel in a single forward pass using a tree-structured attention mask. Accepted tokens are kept; the first rejected token triggers a corrected sample from the target distribution, ensuring exact distributional equivalence. Speedup is governed by the acceptance rate $\alpha$: latency reduction is $\approx K \cdot \alpha / ( 1 + K \cdot c )$ where $c$ is the relative cost of the draft model. Speculative decoding helps when tokens are predictable (code, templated text) and hurts when acceptance rate is low (creative generation, multilingual). See Section 18.1. Senior follow-up. “How do you choose the draft model? What properties make a good draft model for a given target model?” The draft model should be 10-20$\times$ cheaper than the target (parameter count or latency ratio), trained on the same data distribution, and tokenizer-compatible. In practice, self-speculative decoding (using early layers of the target as the draft) avoids the tokenizer compatibility problem and the memory overhead of a second model; EAGLE and Medusa extend this to learned auxiliary heads.

4. What quantization schemes are production-viable today, and what are the precision-quality tradeoffs? Skeleton. INT8 weight-only quantization (e.g., LLM.int8()) is effectively lossless for most models: perplexity degrades $<$1% and throughput improves $\sim$1.5-2$\times$ due to halved weight memory. INT4 (GPTQ, AWQ) is more aggressive: quality degrades noticeably on reasoning tasks unless per-group quantization is used (128-token groups typically). Activation quantization (W8A8, W4A8) enables integer matrix multiplies and is the path to $4 \times$ throughput, but requires careful handling of activation outliers (SmoothQuant absorbs them into weight scales). See Section 19.1. Senior follow-up. “What is an activation outlier and why does it break naive INT8 activation quantization?” Certain dimensions of the residual stream grow to magnitudes $100 \times$ the typical value during training (a phenomenon amplified by depth and model scale). Naive per-tensor quantization rounds small activations to zero when the scale is set by the outlier; SmoothQuant migrates the scaling difficulty from activations to weights by multiplying the outlier dimension in activations by $1 / s$ and the corresponding weight row by $s$ before quantization.

5. Describe continuous batching and why it is preferred over static batching for LLM serving. Skeleton. Static batching waits until all sequences in a batch complete before accepting new requests; the batch is padded to the longest sequence, wasting compute on pad tokens and blocking new requests until the slowest sequence finishes. Continuous batching (iteration-level scheduling) inserts new requests into the batch at each decode step, immediately after any sequence finishes. This improves GPU utilization from $\sim$30-50% to $\sim$80%+ in typical workloads by eliminating both padding waste and request queuing delay. See Section 18.2. Senior follow-up. “How does prefix caching interact with continuous batching, and what are the memory management implications?” Prefix caching reuses KV blocks for shared prompt prefixes (e.g., a

system prompt shared across requests); with continuous batching, blocks must be reference-counted and reclaimed when all sequences sharing a prefix complete. Eviction policies (LRU, radix-tree-based) determine cache hit rate; a senior answer notes that prefix cache hit rate is the primary factor driving effective throughput on chatbot workloads with long system prompts.

6. A user asks why their 7B model is slower per token at batch size 32 than at batch size 1, even though the GPU utilization is higher. Diagnose and explain. Skeleton. At batch size 1, generation is memory-bandwidth-bound: each step loads model weights once and performs one matrix-vector multiply. At batch size 32, the weight loading is amortized over 32 sequences (good), but KV cache loading scales linearly with batch size and sequence length - at large $T$, KV cache reads dominate over weight reads, and the GPU is still memory-bound but now on KV traffic rather than weight traffic. Additionally, at batch size 32 with long sequences the effective arithmetic intensity of the attention step grows only sublinearly, while total data movement grows linearly. See Section 15.1. Senior follow-up. “At what batch size does an attention kernel transition from memory-bound to compute-bound, and how does head dimension affect this crossover?” The crossover occurs when FLOPs per attention step ($\approx 2 \cdot B \cdot T \cdot d$) divided by bytes loaded ($\approx 2 \cdot B \cdot T \cdot d + B \cdot d^{2}$) exceeds the hardware ridge point ($\sim$300 on H100). For small head dimensions, the crossover batch size is large (hard to reach); for larger head dimensions, it is reached earlier. Flash Attention exploits this by fusing tiles to reduce HBM round trips regardless of batch size.

B.3 Alignment and Evaluation Questions

1. What does SFT teach a model, and why is it insufficient for aligning behavior to human preferences? Skeleton. Supervised Fine-Tuning teaches the model to imitate the distribution of demonstration data: it maximizes log-likelihood over high-quality responses. This improves instruction-following and surfacelevel helpfulness but does not optimize for human preference between responses, cannot teach the model to refuse harmful requests it was not explicitly shown refusing, and introduces hallucination pressure when the model is forced to complete responses in domains where the training data contains confident but incorrect completions. The model learns what was said, not what is preferred. See Section 26.1. Senior follow-up. “In practice, SFT often degrades factual accuracy relative to the pretrained base model. What is the mechanism?” SFT data curators select fluent, complete-sounding responses; the model learns to produce confident completions regardless of factual grounding. This overrides the base model’s calibration, which learned uncertainty from noisy pretraining data. A senior answer distinguishes data-side mitigations (include responses that hedge, refuse, or express uncertainty) from model-side mitigations (KL regularization toward the base model during SFT).

2. Explain the RLHF pipeline: what does the reward model learn, and how is PPO used to optimize it? Skeleton. Human raters compare pairs of responses; a Bradley-Terry model fits a scalar reward function $r_{\phi}$ to predict which response is preferred. The policy $\pi_{\theta}$ is then optimized with PPO to maximize $\mathbb{E} [ r_{\phi} ( x , y ) ] - \beta \cdot$ KL$( \pi_{\theta} \| \pi_{\text{ref}} )$, where the KL term prevents the policy from collapsing to reward-hacking outputs that are out-of-distribution for the reward model. PPO requires maintaining the current policy, the old policy (for importance sampling), the reference policy (for KL), and the value function - four model copies in memory simultaneously. See Section 27.1 and Section 27.2.

Senior follow-up. “What is reward overoptimization, how do you detect it, and how do you mitigate it?” Overoptimization occurs when the policy finds outputs that score high under $r_{\phi}$ but are not actually preferred by humans - a proxy gaming failure. Detection: track gold human preference win-rate alongside $r_{\phi}$ score; divergence between the two is the signal. Mitigation: lower $\beta$ (less KL regularization) is the wrong direction; increase $\beta$, reduce PPO steps per batch, or use an ensemble of reward models to make reward hacking harder.

3. What is DPO, and what is its main advantage over PPO for preference alignment? Skeleton. Direct Preference Optimization reparameterizes the RLHF objective to eliminate the explicit reward model: it shows that the optimal policy under the KL-constrained reward maximization objective satisfies $r ( x , y ) = \beta \log^{\frac{\pi ( y | x )}{\pi_{\text{ref}} ( y | x )}} + \beta \log Z ( x )$, allowing the reward to be implicitly defined by policy log-ratios. The training loss becomes a binary cross-entropy over preferred/rejected pairs. DPO removes the value network and eliminates the rollout generation step, reducing memory from 4 model copies (PPO) to 2 (policy + reference). See Section 27.2. Senior follow-up. “DPO is often observed to reduce output diversity and produce shorter responses than PPO. What is the mechanism, and how do variants like SimPO or IPO address it?” DPO’s gradient pushes the policy to increase the likelihood of chosen responses and decrease the likelihood of rejected responses; if the chosen responses are uniformly shorter or less varied, the policy collapses toward that mode. SimPO adds a length normalization term to the reward; IPO uses an identity link instead of the Bradley-Terry sigmoid, which reduces over-confidence on pairs that are nearly tied in quality.

4. What is GRPO, and why does it matter that it eliminates the value network? Skeleton. Group Relative Policy Optimization replaces the PPO value function with a group-level baseline: for each prompt, $G$ completions are sampled and the reward for each is normalized by the group mean and standard deviation, yielding a self-contained advantage estimate without a learned critic. This removes the fourth model copy (value network) from memory, reducing peak VRAM by $\sim$25% relative to PPO and eliminating the value function’s training instability. GRPO is the training algorithm behind DeepSeek-R1’s reasoning-chain optimization. See Section 28.2. Senior follow-up. “GRPO maintains an old policy for importance sampling. How does the number of reuse steps of old-policy rollouts interact with training stability?” Reusing rollouts for more than $\sim$2-4 gradient steps increases the importance-sampling ratio $\pi_{\theta} / \pi_{\text{old}}$ beyond the PPO clip range ($\epsilon \approx 0 . 2$), causing gradient variance to explode. GRPO clips the ratio with the same $\epsilon$ as PPO but clips at the group level; increasing the number of reuse steps requires widening $\epsilon$ or shrinking the learning rate, both of which slow convergence.

5. What is RL from Verifiable Rewards (RLVR), and why does it produce more reliable training signal than a learned reward model? Skeleton. RLVR trains on tasks where correctness can be checked without human judgment: math problems with known solutions, code that passes unit tests, formal proofs verified by a checker. The reward is binary (correct/incorrect) and not subject to reward model overoptimization because there is no proxy to game - a wrong answer simply scores zero. This produces a stable, non-hackable signal but limits training to domains where ground truth is available. DeepSeek-R1 and OpenAI’s o-series models both use RLVR as a component of their reasoning capability. See Section 28.1. Senior follow-up. “What failure modes can still occur with binary verifiable rewards, even though reward hacking is prevented?” Process hacking: the model finds solutions that satisfy the verifier without gen-eralizing (e.g., memorizing problem templates, outputting correct final answers via invalid intermediate

steps). Length exploitation: the model learns that longer chains of reasoning are rewarded due to correlation in the training set, not because length causally improves correctness. A senior answer suggests process reward models or step-level verification as mitigations.

6. How do you detect and mitigate benchmark contamination when evaluating a model? Skeleton. Contamination occurs when benchmark examples appear in pretraining data, directly or via near-duplicates; a model can score high by retrieval rather than generalization. Detection strategies include: n-gram overlap between benchmark test sets and pretraining corpora, membership inference attacks (testing whether the model assigns higher likelihood to test examples than held-out examples from the same distribution), and canary insertion (embedding synthetic labeled examples at data collection time and later checking if the model recalls them). Mitigation is primarily data-side: deduplicate pretraining data against known benchmarks before training. See Section 25.1. Senior follow-up. “If you cannot access the pretraining corpus to check for contamination, what evaluation design choices reduce contamination risk?” Use held-out private benchmarks not released publicly; generate new examples programmatically at evaluation time (e.g., LiveCodeBench, FrontierMath); or use behavioral probes that require reasoning steps rather than answer recall. Chatbot Arena sidesteps contamination entirely by using open-ended human preferences on novel prompts. See Chapter 24.

7. Why is perplexity on a held-out set a poor proxy for downstream task performance, and what should you measure instead? Skeleton. Perplexity measures the model’s ability to predict the next token averaged over a reference corpus; it is sensitive to tokenization, domain distribution, and corpus quality rather than to task-relevant capabilities. A model with lower perplexity can score worse on MMLU, HumanEval, or instruction-following benchmarks if the pretraining corpus is high-quality but domain-mismatched to the evaluation tasks. For alignment-stage evaluation, win-rate against a reference model on human preference judgments (or GPT-4 as a proxy judge) correlates better with real-world utility than perplexity. See Section 23.1. Senior follow-up. “GPT-4-as-judge is widely used but has known biases. Name two and describe how to mitigate them.” Position bias (GPT-4 favors responses in the first position when presented with two choices): mitigate by randomizing order and averaging. Verbosity bias (longer responses are judged as better regardless of quality): mitigate by constraining response length or instructing the judge to penalize unnecessary verbosity. A senior answer also notes self-enhancement bias (GPT-4 favors outputs similar to its own training distribution) and suggests using multiple judge models or human calibration.