Chapter 1 The LLM Systems Interview

1.1 What Companies Actually Ask

1.1.1 Design-an-LLM-System Questions

You sit down across from the interviewer. After two minutes of introductions, the question lands: “Design a serving system for a code-completion product with p95 time-to-first-token under 200 milliseconds.” You have 45 minutes. This is the open-ended design question, and it is the center of gravity for senior and staff-level LLM system roles.

Design questions ask you to build an LLM system end to end: from hardware budget to serving scheduler, from a one-line product requirement to a set of motivated architectural choices. They are open-ended on purpose. The interviewer is not looking for a canonical architecture; they are watching how you structure ambiguity, which constraints you surface first, and whether your decisions are driven by actual hardware physics or by patternmatching to things you have read.

The question has three phases that map directly to the framework in Chapter 2. First, you clarify requirements: TTFT SLO, throughput target, quality bar, and cost ceiling-because the answer to “which model size?” and “which parallelism strategy?” changes completely depending on these. Second, you sketch the end-to-end pipeline: tokenization, prefill (compute-bound, KV cache written), decode (memory-bandwidth-bound, KV cache read each step), and the serving scheduler that batches concurrent requests. Third, you identify the binding constraint-almost always KV cache pressure rather than raw FLOPs-and propose optimizations as targeted responses to that constraint rather than as a list of things you have memorized.

Staff-level and principal-level design questions add a second layer: the interviewer changes a constraint mid-answer. “What if context length doubles?” “What if you need to serve five model variants behind the same endpoint?” “What if cost-per-token becomes the primary SLO?” A candidate who derived their numbers from first principles adapts in seconds. A candidate who memorized an architecture has to start over. This is why the framework matters more than any specific configuration.

The vocabulary these questions test is introduced in Chapter 2 and derived in Parts IV through VII: arithmetic intensity, memory-bandwidth-bound versus compute-bound regimes, KV cache footprint per sequence per layer, batch-size-versus-latency trade-off, continuous batching, speculative decoding, grouped-query attention.

Every strong answer to a design question is derivable from the constraints on the whiteboard; the interviewer can always verify this by changing one constraint and watching what happens.

How This Shows Up

• “Design a production serving stack for a 7B chat model targeting p95 TTFT under 300 ms at 5,000 QPS.” A strong answer opens with requirement clarification (interactive or batch use? single or multi-tenant?), sketches prefill and decode as separate boxes with different bottlenecks, estimates KV cache memory at a target batch size, identifies the batch-size constraint, and proposes continuous batching before discussing any optimization. Chapter 2 covers the framework; Chapters 15 and 18 cover the arithmetic and serving optimizations.

• “Design the training infrastructure for a 70B dense model on a 512-GPU cluster.” A strong answer sizes the memory budget (parameters, gradients, optimizer states), picks a parallelism combination justified by the hardware topology (tensor-parallel within node, pipeline-parallel or ZeRO-3 across nodes), estimates the pipeline bubble, and names where batch size is being consumed as a resource. Chapter 12 covers parallelism strategies; Chapter 13 covers the FLOP and memory arithmetic.

Key Takeaways

• Open every design answer with requirements-TTFT SLO, throughput target, quality bar, cost ceilingbefore naming any component.

• In serving, the binding constraint is almost always KV cache memory pressure, not FLOPs; in training it is usually communication overhead.

• Interviewers change constraints mid-answer to distinguish first-principles reasoning from memorized configurations-derive, do not recall.

• Chapter 2 is the procedural skeleton; the technical chapters (Parts IV-VII) provide the arithmetic to fill it in.

1.1.2 Deep-Dive Component Questions

Twenty minutes into the design question-or sometimes as the entire interview-the interviewer pivots: “Let’s go deeper on speculative decoding. Walk me through the acceptance criterion. Why does that sampling procedure give you an exact sample from the target distribution?” This is the deep-dive format: narrow, mechanismlevel, and unforgiving about gaps between vocabulary and understanding.

Deep-dive questions test whether you understand the mechanism, not just the outcome. “Grouped-query attention reduces the KV cache” is something you can read in an abstract. “GQA replaces $N$ independent key-value head pairs with $G$ shared groups, reducing KV cache memory by a factor of $N / G$, which allows a proportionally larger batch size on the same hardware” is what the question is actually asking for. The follow-up will probe the second-order consequence: a larger batch size raises arithmetic intensity in the MLP layers toward the compute-bound regime-while attention remains memory-bandwidth-bound regardless-so GQA changes where the bottleneck sits at different batch sizes.

The components that appear most often in deep dives map directly to the technical parts of this book:

• Architecture details: attention variants (GQA, MLA, sliding-window attention), MoE routing, normalization and positional embedding choices-Parts II and III.

• GPU and kernel mechanics: memory hierarchy, arithmetic intensity, operator fusion, FlashAttention-Part IV.

• Parallelism: all-reduce versus all-gather, ZeRO stages and their communication cost, pipeline bubbles, tensor versus pipeline trade-offs-Part V.

• Inference systems: KV cache sizing, continuous batching, speculative decoding acceptance criterion, quantization precision tiers and their accuracy risk-Part VII.

• Evaluation: benchmark contamination, evaluation harness design, metric selection for instruction-following and reasoning tasks-Part IX.

The depth expectation scales with seniority. A junior screen asks for a correct description of the mechanism. A senior screen asks for the trade-off and the failure mode. A staff or principal screen asks for the case where the standard answer is wrong-and how you would detect it before shipping. Calibrate your answer to the level you are interviewing for; going too shallow loses the offer, but going deep and then being wrong about the second-order consequence is worse.

A deep-dive answer has two moves: state the mechanism precisely, then name the second-order consequence that the mechanism creates or removes.

How This Shows Up

• “Explain why token generation is memory-bandwidth-bound even at large batch sizes.” A strong answer works from arithmetic intensity: each decode step processes one new token per sequence, collapsing the weight matrix multiplications to matrix-vector products with intensity proportional to batch size divided by hidden dimension-a number almost always below the GPU’s compute-bandwidth ridge point regardless of batch size. The derivation lives in Chapter 15.

• “How does ZeRO stage 3 differ from ZeRO stage 1 in memory savings and communication cost?” A strong answer traces what is sharded at each stage (optimizer state at stage 1, gradients at stage 2, parameters at stage 3), quantifies the communication increase (roughly $3 \times$ parameter bytes per step at stage 3 versus $1 \times$ at stage 1), and notes that overlapping the all-gather with the forward pass is what makes stage 3 viable at scale. Chapter 12 covers this in full.

Key Takeaways

• Deep dives test mechanism, not vocabulary: “it reduces memory” is the opening of an answer, not the answer.

• State the second-order consequence-what the mechanism enables, constrains, or shifts-to move from a passing answer to a strong one.

• Seniority raises the bar from correct description to trade-off to failure mode; calibrate accordingly.

• The technical chapters in Parts II-IX are organized around the component questions that appear most frequently in frontier-lab interviews.

1.2 Roles That Require LLM Systems Knowledge

1.2.1 LLM Infrastructure and Training Engineer

Picture the engineer who gets paged because a 512-GPU training run has stalled at step 47,000. The job is not to design a new architecture; it is to figure out why GPU 203 is 40% slower than the rest-whether the culprit is a straggling all-reduce, a pipeline bubble, or a dead NVLink lane. This role exists at every frontier lab and most serious AI infrastructure teams, and its interview goes deep into hardware and distributed systems.

The infra and training engineer owns the layer between model code and the cluster. Their primary vocabulary is hardware topology-NVLink bandwidth within a node versus InfiniBand across racks-collectivecommunication primitives (all-reduce, all-gather, reduce-scatter), and the three axes of model parallelism: data, tensor, and pipeline.

No single parallelism strategy dominates. Data parallelism scales compute linearly but does nothing for per-GPU memory; ZeRO stage 3 (FSDP) reclaims memory by sharding parameters and optimizer states across workers, but it raises total communication cost to roughly three times the parameter count per step. Tensor parallelism demands ultra-fast intra-node links and is practical only up to the number of GPUs per machinetypically eight-because going further triggers the slower inter-node fabric. Pipeline parallelism crosses machines cheaply but introduces a bubble proportional to stages divided by micro-batches. Production runs combine all three: tensor-parallel within a node, pipeline-parallel or FSDP across nodes, data-parallel at the outermost level. We work through the arithmetic behind each strategy in Part III.

Batch size is not just a hyperparameter here-it is a scarce resource. Both data parallelism and pipeline parallelism consume it, so a candidate who can articulate that trade-off signals genuine systems fluency. The other half of the role is resilience: silent GPU corruption, checkpoint cadence, and automatic-restart logic. At scale, hardware failures are routine, not exceptional.

The training engineer’s job is to convert cluster-hours into useful gradient steps, not communication stalls or idle GPUs.

How This Shows Up

• “How would you distribute a 70B-parameter model across a 64-GPU cluster for training?” A strong answer picks a parallelism combination, justifies it by the memory and bandwidth constraints of the hardware (8-GPU nodes connected by NVLink internally, InfiniBand across nodes), estimates the bubble or bandwidth overhead, and names where batch size is being spent as a resource. The derivations live in Part III.

• “Walk me through how FSDP reduces optimizer-state memory compared to naive DDP.” A strong answer traces the three ZeRO stages-shard optimizer state (stage 1), then gradients (stage 2), then parameters (stage 3)-explains why stage 3 raises communication to three times the parameter count, and notes that the overlap of communication with computation is what makes it viable in practice. Chapter 8 covers this in full.

Key Takeaways

• Parallelism is a combination problem: tensor-parallel within a node, pipeline-parallel or ZeRO-3 across nodes, data-parallel at the outermost level.

• Batch size is a scarce resource that both data parallelism and pipeline parallelism consume; surfacing this trade-off signals systems maturity.

• Communication is the bottleneck, not compute: on modern clusters, the training engineer reasons about memory bandwidth and collective-communication cost more than raw FLOP counts.

• Hardware failures at scale are routine-checkpoint strategy and fault tolerance are not afterthoughts.

1.2.2 LLM Applied/Product Engineer

The applied engineer takes a trained or fine-tuned model and turns it into a product that ten thousand users hit every minute. Their interview is lighter on distributed-training arcana and heavier on inference economics: how fast does the model respond, how much does it cost to serve, and what breaks first when traffic spikes?

The core abstraction the applied engineer must own is the two-phase structure of LLM inference: prefill (encoding the prompt in parallel, compute-bound) and generation (emitting tokens one by one, memory-bound). During generation, each step reads all model parameters plus the entire KV cache from high-bandwidth memory; arithmetic intensity drops to roughly one FLOP per byte regardless of batch size, because every sequence maintains its own KV state. That ceiling is set by memory bandwidth, not the GPU’s peak FLOP count. We derive this bound in Chapter 9.

From that ceiling follow all practical serving decisions. To improve throughput, you batch requests togetherlarger batches amortize the parameter-read cost across more tokens, pushing arithmetic intensity toward the compute-bound regime. To reduce memory pressure, you shrink the KV cache: grouped-query attention uses fewer key-value heads, multi-head latent attention projects keys and values into a lower-dimensional space, and local attention restricts each layer to a sliding context window. Quantization (FP8 or INT4 weights) cuts parameter-read time directly. Speculative decoding uses a cheap draft model to generate candidate tokens that the large model verifies in one parallel prefill pass-since prefill is fast and parallel, this can nearly double effective generation throughput with no accuracy loss.

The applied engineer also needs fluency in batching dynamics under live traffic: continuous batching so the GPU never idles waiting for a fixed batch to fill, KV cache eviction policies when memory is exhausted, and the latency-throughput curve that determines where to set serving SLOs for a given user experience target.

Inference at scale is a memory-bandwidth problem wrapped in a queueing problem.

How This Shows Up

• “Design a serving system for a 7B chat model that needs p95 latency under 500ms at 5,000 QPS.” A strong answer estimates peak memory (parameters plus KV cache at the target batch size), checks whether the system is memory-bound or compute-bound at that batch size, proposes horizontal scaling or model sharding if a single GPU cannot hold the load, and discusses continuous batching. The quantitative framework lives in Chapter 9.

• “What are the trade-offs between serving a 70B model in FP16 versus INT4?” A strong answer covers: memory reduction (roughly $4 \times$ versus the FP16 baseline), latency improvement proportional to memory reduction in the memory-bound regime, and accuracy risk driven by outlier activations in large models. Chapter 10 covers quantization schemes in depth.

Key Takeaways

• Generation is memory-bound: the bottleneck is reading weights and KV cache from HBM, not performing FLOPs.

• KV cache dominates serving memory; shrinking it via GQA, quantization, or local attention is the highestleverage optimization.

• Latency and throughput trade against each other: smaller batches favor latency, larger batches favor throughput-know where your product’s SLO sits on that curve.

• Speculative decoding can double throughput at no accuracy cost; it is the canonical example of exploiting the asymmetry between prefill speed and generation speed.

1.2.3 Research Engineer vs. Research Scientist

At a small startup these roles blur into one person who trains models and writes papers the same week; at a frontier lab they can be two distinct career tracks with different interview loops and different hiring bars. Getting the distinction wrong-showing up to a research engineer screen expecting research scientist questions, or vice versa-is a fast path to a misaligned conversation.

The research engineer (RE) is an engineer first. Their core job is to implement, debug, and scale research ideas: modify training loops, wire up new data pipelines, profile GPU utilization, and ensure that an experiment proposed by a scientist actually runs correctly on a thousand-GPU cluster. The RE interview tests implementation depth-can you trace through FSDP, explain why a training loss spiked at step 30,000, or spot a numerical instability in a custom kernel? Systems judgment matters: the RE who cannot reason about memory budgets and parallelism trade-offs will block every experiment they try to run.

The research scientist (RS) is a researcher first. Their core job is to generate hypotheses about what might make models better-new architectures, training objectives, data mixtures, alignment strategies-and to design experiments that test those hypotheses rigorously and efficiently. The RS interview tests scientific judgment: can you critique an ablation design, identify confounders, propose a controlled experiment to isolate the effect of a curriculum-learning schedule, or interpret a result that partially contradicts your hypothesis? Quantitative rigor matters: the RS who cannot design a fair comparison or read a scaling-law plot will produce papers that do not hold up.

Both roles require LLM systems fluency, just at different depths. An RE who cannot reason about inference latency or training parallelism will be blocked on every implementation. An RS who cannot read a profiler trace or estimate memory budgets will propose experiments that are not runnable. The technical chapters in this book-on parallelism, inference, and evaluation-are directly relevant to both: the RE needs implementation-level depth, the RS needs enough to communicate clearly with the engineering team and to scope experiments that fit within a given compute budget.

The interview distinguishes RE from RS by asking whether you built the system or designed what the system should test.

How This Shows Up

• “Tell me about a training run you debugged from first symptom to resolution.” This is a research engineer question. A strong answer traces symptoms (loss spike, low GPU utilization, OOM), lists hypotheses, describes diagnostic steps (profiler, log inspection, checkpoint bisection), and explains the fix. Vague or high-level answers signal limited hands-on implementation exposure.

• “Propose an experiment to determine whether grouped-query attention hurts model quality on long-context tasks.” This is a research scientist question. A strong answer identifies confounders (training sequence length, model size, evaluation metric), defines a clean baseline, justifies the compute budget, and discusses what a null result would mean. Chapter 5 covers grouped-query attention; Chapter 12 covers evaluation design.

Key Takeaways

• RE interviews test implementation depth: profiling, debugging, and hands-on experience with distributed training and inference systems.

• RS interviews test research judgment: experimental design, hypothesis generation, and interpreting ambiguous or contradictory results.

• Both roles require LLM systems fluency; the difference is depth of implementation versus depth of scientific reasoning.

• If you are unsure which loop you are in, ask the recruiter before preparing-the prep is genuinely different.