Contents
Prefacevi
How to Use This Bookvii
For Candidates Preparing for Interviewsx
For Engineers Building LLM Systemsxiii
Notation and Symbolsxvi
Acknowledgmentsxxi
About the Authorxxii
IOverview and Interview Landscape1
Chapter 1 The LLM Systems Interview2
1.1What Companies Actually Ask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
1.2Roles That Require LLM Systems Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . .4
Chapter 2 How to Approach an LLM System Design Question8
2.1A Repeatable Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
2.2Trade-Off Axes You Must Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
2.3Common Pitfalls in Answers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
IIFoundations: Transformer Architecture and Training Signals16
Chapter 3 Transformer Architecture Interview Essentials17
3.1The Baseline Decoder-Only Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
3.2Normalization Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
3.3Activation Functions in Modern LLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28
3.4Position Encodings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
Chapter 4 Hyperparameters You Will Be Asked to Justify40
4.1Width and Depth. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40
4.2Attention Head Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45
4.3Vocabulary Size and Tokenization Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . .51
4.4Regularization in Pre-Training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57
Chapter 5 Stability Tricks in Large-Scale Training63
5.1Softmax Instabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
5.2Numerical Precision and Training Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69
IIIMixture of Experts and Sparse Architectures76
Chapter 6 Why MoE Is Now Standard in Frontier Systems77
6.1Dense vs. Sparse Models for Fixed FLOPs . . . . . . . . . . . . . . . . . . . . . . . . . . . .77
6.2Routing: The Heart of Every MoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82
Chapter 7 MoE Training and Systems Considerations89
7.1Fine-Grained and Shared Experts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89
7.2Load Balancing and Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94
7.3Expert Parallelism Gotchas for Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
IVGPUs, Kernels, and Single-Device Performance107
Chapter 8 GPU Architecture for LLM Engineers108
8.1The Compute and Memory Hierarchy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108
8.2Arithmetic Intensity and the Roofline Model . . . . . . . . . . . . . . . . . . . . . . . . . . .114
Chapter 9 Making a Single GPU Go Fast121
9.1Reducing Memory Movement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121
9.2Exploiting Memory Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126
9.3Lower Precision and Quantization-Aware Training . . . . . . . . . . . . . . . . . . . . . . . .133
Chapter 10 Writing and Benchmarking Custom Kernels139
10.1 Benchmarking and Profiling Discipline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .139
10.2 Implementing Kernels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145
10.3 FlashAttention as a Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152
VDistributed Training: Parallelism at Scale159
Chapter 11 Multi-GPU and Multi-Node Fundamentals160
11.1 Interconnect Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .160
11.2 Collective Communication Primitives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165
11.3 The Software Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171
Chapter 12 Parallelism Strategies178
12.1 Data Parallelism and ZeRO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .178
12.2 Model Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184
12.3 Activation and Sequence Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190
12.4 Putting It All Together: 3D and 4D Parallelism. . . . . . . . . . . . . . . . . . . . . . . . .196
VIScaling Laws and Training Economics203
Chapter 13 Predictable Scaling for Interview Answers204
13.1 The Three Canonical Scaling Laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .204
13.2 Using Scaling Laws to Make Design Decisions. . . . . . . . . . . . . . . . . . . . . . . . .210
Chapter 14 Chinchilla and Beyond216
14.1 Compute-Optimal Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .216
14.2 Inference-Aware Token-to-Parameter Ratios . . . . . . . . . . . . . . . . . . . . . . . . . . .222
14.3 Maximal Update Parameterization (muP). . . . . . . . . . . . . . . . . . . . . . . . . . . .228
VIIInference Systems234
Chapter 15 The Inference Workload235
15.1 Prefill vs. Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .235
15.2 Latency, Throughput, and Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .240
Chapter 16 Reducing the KV Cache247
16.1 Attention Variants for Cheaper Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . .247
16.2 Cross-Layer and Local Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252
Chapter 17 Going Beyond the Transformer for Inference258
17.1 State-Space and Linear-Attention Models. . . . . . . . . . . . . . . . . . . . . . . . . . . .258
17.2 Non-Autoregressive Generation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .264
Chapter 18 Speculative Decoding and Serving Optimizations270
18.1 Speculative Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .270
18.2 Serving System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .276
Chapter 19 Compression Techniques for Deployment283
19.1 Quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .283
19.2 Pruning and Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .288
VIIIData: The Real Differentiator294
Chapter 20 Pre-Training Data Pipelines295
20.1 Where Training Data Comes From . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .295
20.2 Evolution of Open Pre-Training Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . .300
20.3 Legal and Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .306
Chapter 21 Data Filtering and Deduplication Algorithms312
21.1 Quality Filtering Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .312
21.2 Targeted Filtering Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .318
21.3 Deduplication at Scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .323
Chapter 22 Mid-Training and Post-Training Data330
22.1 Instruction and Chat Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .330
22.2 Long-Context and Domain Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .336
22.3 Data Quality Heuristics That Matter in Interviews . . . . . . . . . . . . . . . . . . . . . . . .342
IXEvaluation349
Chapter 23 Designing an Evaluation350
23.1 Goals of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .350
23.2 Metrics and Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .355
Chapter 24 Benchmarks You Must Know362
24.1 Knowledge and Reasoning Benchmarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . .362
24.2 Instruction Following and Chat Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . .367
24.3 Agentic and Safety Benchmarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .374
Chapter 25 Validity, Contamination, and Real-World Use380
25.1 Train-Test Overlap and Contamination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .380
25.2 Real-World Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .385
XAlignment and Post-Training391
Chapter 26 Supervised Fine-Tuning392
26.1 What SFT Can and Cannot Teach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .392
26.2 SFT Data in Practice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .398
Chapter 27 Preference-Based Alignment (RLHF)405
27.1 The RLHF Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .405
27.2 Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .411
27.3 Pitfalls of RLHF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .417
Chapter 28 Reinforcement Learning from Verifiable Rewards423
28.1 From RLHF to RLVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .423
28.2 Policy Gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .429
28.3 Case Studies in Reasoning Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .435
XIEnd-to-End System Design Drills441
Chapter 29 Building a Production LLM Serving Stack442
29.1 Reference Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .442
29.2 Capacity Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .448
Chapter 30 Designing a Pre-Training Run from Scratch454
30.1 From Compute Budget to Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .454
30.2 Operations and Failure Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .460
Chapter 31 Designing a Fine-Tuning and Alignment Pipeline466
31.1 Scoping the Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .466
31.2 Operational Concerns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .472 Appendix A Napkin Math for LLM Interviews478 A.1Parameter Counts and Memory Footprints . . . . . . . . . . . . . . . . . . . . . . . . . . . .478 A.2FLOPs per Forward and Backward Pass . . . . . . . . . . . . . . . . . . . . . . . . . . . . .479 A.3KV-Cache Sizing and Latency Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . .481
Appendix BCommon Interview Questions483 B.1Architecture and Training Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .483 B.2Inference and Serving Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .485 B.3Alignment and Evaluation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .487
Appendix C Checklists for System Design Answers490 C.1The 10-Minute LLM System Design Checklist . . . . . . . . . . . . . . . . . . . . . . . . . .490 C.2The Pre-Training Run Readiness Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . .491 C.3The Inference Deployment Readiness Checklist . . . . . . . . . . . . . . . . . . . . . . . . .492
Appendix D Further Reading495 D.1Canonical Papers by Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .495 D.2Blogs, Talks, and Engineering Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .501