Chapter 2 How to Approach an LLM System Design Question

2.1 A Repeatable Framework

2.1.1 Clarify Requirements (Latency, Throughput, Quality, Cost)

The interview prompt arrives-“design an LLM-powered customer support system.” Before a single component is named, a strong candidate pauses and asks four questions. At the senior and staff level, interviewers watch specifically for this pause; candidates who skip it and dive straight into architecture are signaling that they have memorized solutions rather than learned to think.

The four axes that every LLM system lives on are latency, throughput, quality, and cost. These are not platitudes; each has a concrete definition that will drive every downstream decision you make.

Latency has two distinct flavors in LLM serving. Time-to-first-token (TTFT) measures how long a user waits before any output appears-this is dominated by the prefill step, which processes the full prompt in one parallel forward pass and is compute-bound. Per-token latency (decode latency) measures how quickly subsequent tokens arrive-this is dominated by the autoregressive decode loop, which is memory-bandwidth-bound because model weights and the KV cache must traverse the memory hierarchy once per generated token. Conflating TTFT with decode latency is one of the most common interview errors; they respond to entirely different optimizations. We derive the arithmetic behind both in Chapter 7.

Throughput measures tokens generated per second across all concurrent users, and it is the number that determines your inference bill at scale. Higher throughput generally requires larger batch sizes, which increases per-request latency. This tension-batching as the lever between latency-optimized and throughput-optimized operating points-is the central trade-off in LLM serving.

Quality sets the floor below which no amount of speed matters. Quality requirements dictate model size, which sets the memory footprint, which in turn constrains the batch size achievable on a given hardware budget. A requirement for GPT-4-class output quality effectively forces you onto a large model and limits your throughput options from the start.

Cost ties the other three together. A 70B model serving a 200 ms TTFT SLO on H100s implies a fundamentally different cost envelope than a 7B model with a two-second tolerance. Surfacing cost early prevents designs that are technically correct but operationally unshippable.

Interviewers want to hear you name these four axes and immediately note which ones are in tension for the specific scenario given. Clarifying requirements is not housekeeping-it is the first technical decision, because every architecture choice is a trade-off between latency, throughput, quality, and cost.

How This Shows Up

• “We’re building a real-time coding assistant. Walk me through how you’d design the serving stack.” A strong answer first asks: What is the TTFT SLO? Is usage interactive or batch? What quality bar? The answer skeleton names TTFT (sub-200 ms is typical for autocomplete), per-token latency, and concurrent user count before touching any component. The technical depth for meeting those SLOs lives in Chapter 7.

• “How would you size the inference fleet for a new chat product launching with 10M daily active users?” A strong answer immediately converts DAU to peak requests per second, estimates average prompt and completion lengths to derive a tokens-per-second demand figure, then works backward to a hardware configuration. Candidates who jump to “use an H100” without this derivation fail the follow-up.

Key Takeaways

• Ask for the latency SLO before naming any architecture; TTFT and per-token decode latency are separate problems with separate solutions.

• Throughput and per-request latency trade against each other via batch size-name this tension explicitly and early.

• Quality requirements set the model size, which sets the memory budget, which constrains achievable batch size and throughput.

• Cost is the constraint that forces a choice between the above; surfacing it in the requirements phase prevents designs that cannot be shipped.

2.1.2 Sketch the End-to-End Pipeline

After requirements are on the board, the next thing an interviewer watches for is whether you can draw the right boxes in the right order. Many candidates jump immediately to optimizations-quantization, speculative decoding, model compression-before they have established the baseline pipeline. That inversion signals patternmatching, not understanding; experienced interviewers catch it within the first minute.

Every autoregressive LLM serving system has two distinct computational phases, and your whiteboard diagram must show both before you say anything else.

The prefill phase receives the raw prompt tokens and processes them in a single forward pass through the model. Because all tokens are known upfront, this pass is fully parallelizable across the sequence dimension-structurally identical to a training-time forward pass. Prefill is compute-bound: given a sufficiently long prompt, the arithmetic intensity is high enough to saturate the GPU’s matrix-multiply units. The outputs of prefill are two things: the next-token logit (from which the first generated token is sampled) and the KV cache-the key and value tensors for every attention layer, written to GPU memory so that decode steps do not recompute them from scratch.

The decode phase generates the response token by token. Each step takes the previously generated token, runs it through the full model, reads the accumulated KV cache from high-bandwidth memory, and samples the next token. Because only one new token is processed per step, the matrix multiplications against the model weights reduce to matrix-vector products-the worst-case scenario for arithmetic intensity. Decode is memorybandwidth-bound: speed is gated by how fast model weights and KV cache can be streamed from HBM, not by the GPU’s compute throughput. We derive the exact bound in Chapter 7.

The sketch should also include the serving layer: a scheduler that accepts incoming requests, forms batches (continuous batching is the industry standard), routes them to prefill and decode workers, and manages KV cache memory. The KV cache is the resource that sets the effective batch size ceiling; its size grows linearly with sequence length, number of attention layers, and number of KV heads. Interviewers at frontier labs expect you to know that KV cache pressure-not FLOPs-is usually the binding constraint on how many concurrent requests you can serve. Chapter 8 covers KV cache sizing arithmetic in full.

Once the baseline pipeline is on the board, you can layer in optimizations in priority order: architecture choices that shrink the KV cache (grouped-query attention, multi-head latent attention), quantization to reduce memory bandwidth pressure, and speculative decoding to hide decode latency. Presenting optimizations before the pipeline is the canonical pitfall; presenting them as motivated responses to the constraints you have just identified is what separates candidates who have built these systems from those who have only read about them.

Draw prefill and decode as separate boxes with separate performance characteristics before you say a single word about optimization.

How This Shows Up

• “Walk me through what happens when a user sends a prompt to a production LLM API.” A strong answer covers tokenization, prefill (compute-bound, KV cache written to HBM), the autoregressive decode loop (memory-bound, KV cache read each step), detokenization, and the serving scheduler that batches concurrent requests. Technical depth on each stage lives in Chapters 7 and 8.

• “Where would you look first if users were reporting high latency?” A strong answer immediately asks which latency-TTFT or per-token-then diagnoses accordingly: high TTFT points to a prefill bottleneck (prompt too long, batch too large, insufficient compute provisioning), while high per-token latency points to a decode bottleneck (KV cache pressure, memory bandwidth saturation, or a batch size that exceeds what the memory hierarchy can sustain).

Key Takeaways

• Prefill (compute-bound) and decode (memory-bound) are distinct phases with different bottlenecks; always draw them as separate boxes.

• The KV cache is the bridge between phases and the primary memory consumer; sizing it is a required step in any capacity plan.

• Continuous batching at the serving layer is the standard mechanism for maximizing GPU utilization across requests with varying lengths and arrival times.

• Present the baseline pipeline first, then optimizations as targeted responses to specific constraints-sequencing signals fluency to interviewers.

2.2 Trade-Off Axes You Must Name

2.2.1 Compute vs. Memory vs. Bandwidth

A design interview turns on the moment you identify the real bottleneck. Most engineers, when asked how to speed up a serving endpoint, jump to “add more GPUs” or “use a smaller model” - both potentially valid, but both bypassing the diagnostic question that unlocks everything else: is the system limited by arithmetic throughput, memory capacity, or memory bandwidth? Those three constraints behave differently, respond to different interventions, and show up in different parts of the stack. Conflating them is one of the most common ways to lose credibility in a systems round.

A GPU has two fundamental performance ceilings. The first is compute throughput, measured in floatingpoint operations per second. The second is memory bandwidth, measured in bytes per second. Any given operation sits in one of two regimes: compute-bound, meaning the arithmetic units are the bottleneck, or memorybandwidth-bound, meaning the time is dominated by streaming data through HBM. The ratio of these two hardware numbers - roughly 300 FLOPs per byte on a current-generation data-center GPU - defines a ridge point. Operations with arithmetic intensity above this ridge are compute-bound; operations below it are memory-bound no matter how many FLOPs they perform. We derive this number precisely in Chapter 8.

Matrix multiplications at large batch sizes land firmly above the ridge. That is why training and prefill are fast: the GPU’s multiplier arrays stay saturated. Token generation is the mirror image. Each decode step processes one new token, which reduces the weight-matrix multiply to a matrix-vector product - arithmetic intensity close to 1 FLOPs/byte, deep in memory-bound territory. This is the root cause of slow generation, and it is structural: no amount of faster arithmetic fixes a problem caused by HBM bandwidth. The intervention has to touch memory movement.

Memory capacity is a third, distinct constraint. KV cache grows with batch size and sequence length; once it exhausts VRAM, you cannot increase the batch regardless of how much arithmetic headroom remains. Techniques like grouped-query attention and quantization are as much about fitting more sequences into a fixed memory budget as they are about raw speed.

When a system feels slow, the first question is always: which ceiling are we hitting? The answer determines whether the fix is a kernel rewrite, a memory layout change, or an architectural trade-off.

How This Shows Up

• “GPU utilization is at 80% but token generation is still slow. Where would you look first?” A strong answer distinguishes compute utilization from memory bandwidth utilization. High compute utilization during generation just means the GPU is busy waiting on HBM reads - the real bottleneck is bandwidth, and the path to improvement is reducing memory traffic, not adding FLOPs. The technical depth for this lives in Chapter 9.

• “A normalization layer consumes 25% of wall-clock time but less than 1% of total FLOPs. How do you speed it up?” The answer requires recognizing that normalization is memory-bandwidth-bound, so FLOP-reduction techniques are irrelevant; the leverage is in cutting parameter loads per call (RMSNorm over LayerNorm) and fusing kernel launches to eliminate round-trips to HBM.

Key Takeaways

• Every LLM operation is either compute-bound or memory-bandwidth-bound; the boundary is set by the hardware’s arithmetic intensity ridge point, not by the number of FLOPs.

• Training and prefill are compute-bound at practical batch sizes; token generation is memory-bandwidth-bound by construction - these require different optimization strategies.

• Memory capacity (fitting the model and KV cache in VRAM) is a third, separate constraint that limits batch size independently of bandwidth or compute.

• TFLOP ratings sell GPUs; memory bandwidth and capacity are the operative constraints for inference engineering.

2.2.2 Quality vs. Latency vs. Cost

Every LLM serving decision eventually compresses to three numbers: how good the output is, how fast it arrives, and how much it costs per token. In practice, you can aggressively optimize two of these; the third tends to move against you. Interviewers at frontier labs want to see that you have internalized this triangle before you draw a single box in your architecture diagram - because candidates who skip it waste fifteen minutes proposing solutions that are technically interesting but wrong for the stated problem.

Latency and throughput are not the same metric, and conflating them is the most common error in LLM serving discussions. Latency is the wall-clock time a single user waits - time-to-first-token (TTFT) for the prompt-encoding phase, then per-token decode latency during generation. Throughput is aggregate tokens generated per second across all concurrent users. These optimize in opposite directions: maximizing throughput requires large batch sizes and full GPU saturation; minimizing latency often means small batches, aggressive prefetching, and accepting lower utilization. A system cannot simultaneously be optimal at both without workload-specific scheduling that separates the two traffic classes.

Quality is the axis that is easiest to accidentally sacrifice. Quantizing weights from BF16 to INT4 can triple throughput; it also measurably degrades accuracy on multi-step reasoning, often in ways that aggregate benchmarks do not surface. Grouped-query attention reduces the KV cache by a factor of eight with negligible benchmark loss - but “negligible on standard benchmarks” is a claim about a distribution of tasks, not a guarantee about your specific workload. Part IX covers how to design evaluations that catch these gaps before they reach production.

Cost is the integrating metric. It is not the same as latency, though the two are correlated. A technique that saves 30% of memory per sequence can double the batch size that fits on a single GPU, halving per-token cost - a much larger lever than shaving a few milliseconds off the decode loop. The arithmetic for a specific model and hardware configuration is in Chapter 15; the key habit is to always convert architectural decisions into a per-token-cost estimate before committing to them.

The vocabulary for this axis in interviews: TTFT (latency of the prefill phase, compute-bound, difficult to compress beyond model size), decode latency (per-token, memory-bandwidth-bound, the target of most serving optimizations), throughput (tokens/second aggregate, optimized by batching and hardware utilization), and costper-token (the business metric that synthesizes all three). Speculative decoding (Chapter 18) is notable precisely because it improves latency without degrading quality - an exact sample from the target distribution at draftmodel speed when the acceptance rate is high.

Name the metric you are optimizing before you propose a solution; an interviewer who has to ask “are you optimizing latency or throughput?” has already registered a gap in your systems thinking.

How This Shows Up

• “We are building a real-time coding assistant. What serving configuration would you recommend?” A strong answer names TTFT and per-token decode latency as the primary metrics, not aggregate throughput; proposes small batch sizes or continuous batching with tight SLOs; discusses the quality floor before deciding quantization aggressiveness. The tradeoff details live in Chapter 19.

• “How would you serve the same 70B model to both an interactive product and an overnight batch pipeline?” A strong answer proposes separate serving paths: the interactive path optimizes latency (smaller batches, possibly speculative decoding), the batch path optimizes throughput and cost (larger batches, more aggressive quantization, relaxed TTFT SLO). A single serving configuration that tries to satisfy both will be suboptimal on both.

Key Takeaways

• Latency and throughput optimize in opposite directions; always establish which one your design is targeting before proposing anything.

• Quality degradation from quantization and architectural shortcuts is real and workload-dependent - aggregate benchmark numbers are an unreliable proxy for production quality on specific tasks.

• Cost-per-token is the integrating metric: batch size, quantization, and model architecture all map onto it differently, and the largest levers are usually in batch size and memory efficiency rather than kernel micro-optimization.

• TTFT, decode latency, and throughput are three distinct quantities; using them interchangeably signals a gap in serving systems understanding.

2.3 Common Pitfalls in Answers

2.3.1 Cargo-Culting Numbers Without Derivation

A candidate writes “140 GB” next to their 70B model box, the interviewer nods, and then asks where that number comes from. The candidate pauses. This moment - memorized figure, no derivation - is one of the most reliable separators between senior candidates and everyone else in LLM system design interviews. It is worse than not citing the number at all, because it signals that you treat estimates as incantations rather than tools.

Virtually every number worth citing in an LLM system interview flows from four first-principles chains: bytes per parameter at a stated dtype, the $6 N D$ approximation for total training FLOPs, the per-token KV cache footprint from layer count and key-value head geometry, and arithmetic intensity as the ratio of FLOPs to memory bytes transferred. These four starting points, combined with the hardware specs on the whiteboard, reproduce almost any published figure in under a minute of algebra. Candidates who own the derivation can update their answer the instant an interviewer changes one input; candidates who memorized the output cannot.

The Chinchilla 20-to-1 token-to-parameter ratio is the most common cargo-cult target. Production scaling teams at multiple major labs have re-derived this number and obtained answers that range from roughly 40- to-1 to nearly 200-to-1, depending on data quality, architecture efficiency, and the serving economics of the final model. Those teams do not inherit a ratio from a paper; they build small proxy models, vary compute allocation across isoFLOP slices, fit the minimum, and read the answer. The derivation lives in Chapter 6; the interview expectation is that you know the analysis exists, can name the variables that shift the answer, and are not surprised when the answer differs from 20.

The danger of a memorized number is not that it is wrong - it is often approximately right. The danger is that you have no idea which assumptions it rests on, so the moment the interviewer changes one you are stuck. “How does your memory estimate change if you switch to FP8 inference?” “What if you add grouped-query attention with eight key-value heads?” “What if context length doubles?” Each of these invalidates a memorized answer while posing no problem for a derived one.

A number you can derive is worth ten you have memorized.

How This Shows Up

• “Estimate the memory footprint of a 70B model during inference with a batch of 32 and a 4,096- token context.” A strong answer starts from bytes per parameter at the stated dtype, then adds the KV cache term - proportional to batch size, context length, number of key-value heads, and layer count - and shows which term dominates. It also notes that switching from BF16 to FP8 halves the parameter footprint but does not change the KV cache unless the cache itself is quantized, and that grouped-query attention directly reduces the second term. The derivation for this lives in Chapter 8.

• “How many H100s do you need to train a 7B model on 1 trillion tokens in two weeks?” A strong answer derives total training FLOPs from the $6 N D$ approximation, converts to GPU-hours using a realistic model FLOP utilization (35-50% is typical in practice), and arrives at a card-count before ever citing a benchmark number. Chapter 5 derives the $6 N D$ rule and calibrates MFU against published runs. The technical depth for both estimates lives in Chapter 5 (training arithmetic) and Chapter 8 (inference arithmetic).

Key Takeaways

• Know the derivation of three core estimates: parameter memory at a stated dtype, training FLOPs from model size and token count, and KV cache memory per token per layer.

• State every assumption before giving a number; when an interviewer changes one, update the answer in thirty seconds.

• The Chinchilla 20-to-1 ratio is a starting point, not a law - production teams re-derive it and get different answers depending on data quality and serving requirements.

• Approximate arithmetic done live and transparently is more persuasive than a precise number with no reasoning attached.

2.3.2 Ignoring the Inference Side of the System

A candidate spends forty minutes on a compelling training pipeline - data mix, tokenizer, architecture choices, parallelism strategy - then closes with “and we serve it with a standard inference server.” The interviewer asks: “What is your expected time-to-first-token at P95?” Silence. This is the second major pitfall: treating inference as a deployment detail rather than a first-class design constraint.

Training is a one-time capital expense. Inference is the operating cost you pay every day for every user, and at production scale it routinely exceeds total training cost within months of a launch. The candidate who designs only for training has optimized the wrong half.

The split between prefill and decode is the central concept. Prefill encodes the full prompt in a single parallel pass - it is compute-bound, uses the GPU efficiently, and determines time-to-first-token. Decode generates one token per step, reading the full set of model weights from GPU memory on every step while performing a tiny amount of arithmetic. For a single sequence, the ratio of FLOPs to bytes transferred is approximately one - far below the hundreds-to-one threshold at which a modern accelerator is compute-saturated. Decode is therefore memory-bandwidth-bound no matter how large the model, and essentially nothing about the serving stack changes that physics.

The KV cache is the primary constraint on serving batch size. Every live sequence occupies a per-layer, perhead cache that grows with context length. As batch size increases, the cache grows proportionally and eventually exhausts GPU memory before compute becomes the bottleneck. This is why grouped-query attention and multi-head latent attention exist: they reduce the bytes stored per sequence, allowing more concurrent sequences and improving throughput. These are not training-time curiosities - they are direct responses to a serving constraint, and the decision to use them must be made before training begins, because it cannot be patched afterward.

This is the deeper trap: architecture decisions made at training time lock in your serving economics. The number of key-value heads, the maximum context length, the parameter dtype - all are fixed before you write a line of serving code, and all directly determine KV cache footprint, achievable batch size, and cost-per-token. Surfacing the inference constraint before committing to an architecture is not premature optimization; it is the correct order of operations.

Any system design that ends with “deploy to an inference server” is incomplete.

How This Shows Up

• “Walk me through serving a 70B model to 10,000 concurrent users.” A strong answer immediately names the latency-versus-throughput trade-off, sizes the KV cache per request, identifies how batch size is constrained by total GPU memory, proposes continuous batching to avoid idle compute between requests, and raises head-structure choices that reduce cache pressure per sequence. The arithmetic lives in Chapter 8.

• “Your architecture team wants to reduce key-value heads from 32 to 8 using grouped-query attention. What does that do to serving costs?” A strong answer derives the KV cache reduction directly: cache per sequence shrinks by $4 \times$, so the maximum batch size at fixed memory roughly quadruples, cutting cost-per-token by roughly the same factor at throughput-limited operating points - while leaving time-to-first-token essentially unchanged. Chapter 8 carries the full derivation. The technical depth for both questions lives in Chapter 8 (inference arithmetic, KV cache sizing, batching strategies).

Key Takeaways

• Name the prefill/decode split early; each phase is bottlenecked differently and therefore has different levers.

• KV cache memory, not parameter count, is usually the binding constraint on serving batch size; size it explicitly before finalizing the architecture.

• Head structure, context length, and dtype are training-time decisions with direct serving-economics consequences - raise these constraints before architecture is finalized.

• State at least one latency SLO and one throughput target before ending any serving discussion; “standard inference server” is not an answer.