Appendix C Checklists for System Design Answers

C.1 The 10-Minute LLM System Design Checklist

This checklist operationalizes the repeatable design framework from chapter 2 for a 45-60 minute interview slot. Each item should produce a concrete number or a named decision. Vague answers do not pass. Skip Phase 3 for purely serving-focused prompts; skip Phase 4 for purely training-focused ones.

Phase 1 - Clarify requirements (2 min)

State the serving SLO: time-to-first-token, inter-token latency, and token throughput (see subsection 2.1.1).
State the compute budget: number of GPUs, GPU-hours available, and cost ceiling per query.
Confirm expected traffic shape: median and peak QPS, context-length distribution.
Confirm quality target: benchmark name and minimum acceptable score.

Phase 2 - Size the model (2 min)

State parameter count and justify it (capability target, inference cost, or Chinchilla ratio; see chapter 14).
Compute inference memory: $2 P$ bytes (bf16) or $0 . 5 P$ bytes (int4) (see section A.1).
State whether the model fits on one node; if not, name the split strategy before proceeding.
Confirm GQA KV-head count ( $N_{\mathrm{kv}} = N / 8$ for $\geq$ 7 B models) and its impact on KV cache budget (see subsection 16.1.1).

Phase 3 - Training system design (3 min)

Compute total training FLOPs: $C = 6 N_{T} P$ ; divide by cluster throughput to get projected wall-clock time (see section A.2).
State the parallelism layout: tensor-parallel degree first (up to the NVLink domain, $\leq 8$ on H100 nodes), then pipeline or ZeRO Stage 3/FSDP across nodes (see subsection 12.4.1).
State the ZeRO stage: Stage 1 is essentially free; escalate to Stage 3 only when per-GPU memory exceeds 80% of HBM (see subsection 12.1.2).
State global batch size relative to the critical batch size and confirm the learning rate schedule terminates correctly (see subsection 14.2.2).
State data: total token count, primary sources, and quality filter choice (see chapter 20 and chapter 21).

Phase 4 - Serving system design (2 min)

Compute KV cache per request at maximum context $S$ and the crossover batch size $B^{\ast}$ (see section A.3).
State batching strategy: continuous batching with paged attention for variable-length traffic (see subsection 18.2.1).
Name any compression applied (int8 weights, int4 AWQ, int8 KV cache) and the expected quality impact (see chapter 19).
State the autoscaling trigger: GPU KV-cache utilization $> 80 \%$ or queue depth above threshold.

Phase 5 - Trade-offs and failure modes (1 min)

Name the dominant bottleneck: compute, memory, bandwidth, or inter-GPU communication (see subsection 2.2.1).
Name the first metric you would instrument: MFU for a training system, TTFT for a serving system.
Name one design decision you would revisit if the bottleneck shifts.

When to escalate

The model does not fit on the stated hardware without a coherent split plan: size memory correctly before proposing any architecture.
The stated batch size exceeds the critical batch size by more than $2 \times$ : the extra parallelism buys nothing; reallocate compute to model scale or data volume.
TTFT SLO is under 500 ms for a 70 B $+$ model with no quantization or speculative decoding: flag the constraint explicitly and propose a mitigation.
Token count is below $20 \times P$ : the run is undertrained; quality claims need explicit justification.

C.2 The Pre-Training Run Readiness Checklist

Run this checklist before submitting a large pre-training job. It operationalizes chapter 12, chapter 13, chapter 14, chapter 20, and chapter 21. Items marked (BLOCK) will cause an incorrect or wasted run if skipped.

Data and tokenization

(BLOCK) Confirm the tokenizer vocabulary matches the one used in any continued pre-training or fine-tuning checkpoints.
(BLOCK) Confirm exact deduplication (URL or document hash) and near-deduplication (MinHash/LSH) have both been applied (see section 21.3).
Confirm quality filter is in place: at minimum, language identification, rule-based heuristics (punctuation, word ratio, length), and a fastText or model-based scorer (see subsection 21.1.2).
Confirm total token count. Chinchilla-optimal minimum is $20 \times P$ ; production runs now target $\geq 100 \times P$ to amortize inference cost (see subsection 14.2.1).
(BLOCK) Verify no evaluation benchmark contamination in the training corpus (see section 25.1).
Confirm multi-epoch token repetition is bounded: diminishing returns accelerate past $\sim$ 4 epochs of any single source.

Memory and parallelism layout

Compute per-GPU training memory: $16 P / ($ DP $\times$ TP $\times$ PP $)$ bytes for weights, gradients, and AdamW state, plus activation memory for the local micro-batch (see section A.1).
(BLOCK) Confirm per-GPU VRAM consumption is below 85% of device HBM capacity.
Set tensor-parallel (TP) degree to the count of NVLink-connected GPUs per node ( $\leq 8$ on H100 DGX nodes); increasing TP beyond one node incurs a large all-reduce penalty (see subsection 12.2.1).
Start with ZeRO Stage 1 (optimizer-state sharding); escalate to Stage 3/FSDP only if per-GPU memory still exceeds 80% of HBM after applying TP (see subsection 12.1.2).
If pipeline parallelism is needed, set the micro-batch count $m$ so that the bubble ratio $( p - 1 ) / m \leq 0 . 05$ where $p$ is the pipeline depth (see subsection 12.2.2).
For long-context runs ( $S > 32$ k): confirm context parallelism or ring-attention is configured to shard activation memory across the sequence dimension (see subsection 12.3.2).

Compute budget and batch size

Compute total training FLOPs: $C = 6 N_{T} P$ ; divide by cluster FLOP/s at target MFU to get projected wall-clock time (see section A.2).
Set global batch size $B \leq B_{\text{crit}}$ (critical batch size). $B_{\text{crit}}$ falls as target loss decreases; fit a scaling law or use $B \approx \sqrt{N_{T} / P} \times d$ as a rough starting point (see subsection 13.2.2).
(BLOCK) Fully specify the learning rate schedule, including the termination point. Cosine schedules cannot be truncated early without restarting from scratch. For a WSD schedule, reserve 10-20% of total steps for the decay phase (see subsection 14.2.2).
If using $\mu$ P, confirm per-layer learning rates are scaled by $1 /$ width and initialization variance by $1 /$ fanin (see subsection 14.3.1).

Monitoring and checkpointing

Confirm MFU instrumentation is in place; target $\geq 0 . 4$ for a well-tuned layout, $\geq 0 . 35$ for runs with significant pipeline or ZeRO Stage 3 overhead.
Log gradient $\ell_{2}$ norm every step. A norm $\gg 1$ within the first 500 steps signals instability before it manifests in the loss curve.
Set checkpoint frequency so that a single-node failure loses at most 30 minutes of compute.
(BLOCK) Run a 1,000-step diagnostic at full cluster size before committing the full job. Verify MFU, loss trajectory, and gradient norm are all as expected.

When to escalate

Estimated per-GPU memory exceeds device HBM at any proposed parallelism layout: do not start; revise the TP/PP/ZeRO configuration.
MFU is below 0.15 in the first 100 steps: communication is dominating compute; inspect NCCL and check for topology mismatches before continuing.
Loss diverges or shows spikes within the first 2,000 steps: check learning rate warm-up, gradient clipping threshold, and data pipeline integrity.
Projected wall-clock time exceeds 90 days at current MFU and cluster size: reconsider model scale or invest in additional cluster capacity before launching.
Token count is below $5 \times P$ : the run is severely undertrained; the resulting checkpoint will underperform Chinchilla-optimal models of the same size.

C.3 The Inference Deployment Readiness Checklist

Run this checklist before promoting an inference endpoint to production traffic. It operationalizes chapter 15, chapter 16, chapter 18, and chapter 19. Use the KV-cache formulas in section A.3 for all sizing steps.

Memory budget

Compute weight memory: $M_{\text{weights}} = 2 P$ bytes (bf16); $0 . 5 P$ bytes (int4 AWQ/GPTQ).
Compute KV cache per sequence at maximum context $S$ : $C_{\mathrm{kv}} = 2 \times L \times N_{\mathrm{kv}} \times H \times S \times 2$ bytes (see section A.3).
(BLOCK) Verify $M_{\text{weights}} + B_{\max} \cdot C_{\mathrm{kv}} \leq 0 . 85 \times$ VRAM at the target maximum batch size.
Confirm GQA is enabled with $N_{\mathrm{kv}} = N / 8$ for models $\geq$ 7 B; this shrinks $C_{\mathrm{kv}}$ by $8 \times$ with $< 1 \%$ accuracy loss at scale (see subsection 16.1.1).
If KV cache memory still dominates at target batch size: evaluate int8 KV quantization (halves $C_{\mathrm{kv}}$ , doubles $B_{\max}$ ) or MLA (see subsection 16.1.2).
Enable paged attention to eliminate KV cache memory fragmentation; reduces effective VRAM waste by 20-40% under variable-length workloads (see subsection 18.2.2).

Latency SLOs and batching

State TTFT and ITL SLO numbers explicitly before configuration (e.g., TTFT $\leq 500$ ms, ITL $\leq 50$ ms) (see subsection 15.2.1).
Compute crossover batch size $B^{\ast} = M_{\text{weights}} / C_{\mathrm{kv}}$ ; throughput gains are superlinear below $B^{\ast}$ and sublinear above it.
Compute decode step time at target batch $B$ : $t_{\text{step}} ( B ) = ( M_{\text{weights}} + B \cdot C_{\mathrm{kv}} ) /$ BW. Confirm $t_{\text{step}} \leq$ ITL SLO.
Use continuous batching for variable-length request streams; it prevents the head-of-line blocking that plagues static batching at mixed context lengths (see subsection 18.2.1).
If TTFT dominates the latency budget and prefill is compute-bound, consider prefill/decode disaggregation or speculative decoding with a small draft model (see subsection 18.1.1).

Quantization and compression

State the quantization scheme and its expected accuracy impact before deploying: int8 weights (LLM.int8) $\approx 0 \%$ quality loss; int4 (AWQ, GPTQ) $\approx 1$ -3% (see section 19.1).
(BLOCK) Benchmark token throughput and task accuracy before and after quantization on the deployment hardware; do not infer from published results on different hardware.
Do not quantize the KV cache without task-specific validation; numerical sensitivity varies significantly by domain.

Serving infrastructure and autoscaling

Provision replica count so the fleet sustains peak QPS with at least $1 . 5 \times$ headroom under a single-replica failure.
Set autoscaling trigger on KV-cache utilization ( $> 80 \%$ ) and request queue depth ( $> 100$ requests at P95); do not scale on CPU metrics alone.
For long-context deployments: confirm context length limits are enforced server-side to prevent KV cache overflows that would silently OOM single requests.

Observability

Instrument all of: TTFT, ITL, token throughput, KV-cache utilization, GPU memory usage, and request queue depth.
Log per-request context length and output length; these are the primary drivers of KV cache pressure and are needed to diagnose latency tail events.
Alert when ITL exceeds $2 \times$ the P50 value for more than 60 seconds: this indicates pathological batch composition or KV cache thrashing.

When to escalate

$B_{\max}$ is below 8 at the target context length: the model is too large for the available hardware at that context; apply quantization or reduce the maximum context limit.
TTFT exceeds $2 \times$ the SLO at all tested batch sizes: evaluate prefill/decode disaggregation or speculative decoding before adding hardware.
KV-cache utilization stays above 90% during normal traffic: add replicas or enforce a tighter maximum concurrency cap.
int4 quantization degrades the primary task metric by more than 5% relative: revert to int8 and add hardware instead of accepting the quality loss.
Any GPU OOM event during normal deployment: tighten $B_{\max}$ -the sizing formula assumes uniform sequence lengths, but real workloads have heavy-tailed context distributions.