For Engineers Building LLM Systems

This book is organized around interview questions, but its technical content is not interview-specific. The arithmetic behind batch-size constraints, the parallelism trade-offs in distributed training, the memorybandwidth math behind KV cache sizing-these are the same calculations you do on the job. This section maps the book’s chapters to the decisions you encounter in production.

When to Reach for This Book

You are sizing a serving fleet for a new model Start with Appendix A (KV cache formula, memory footprint) and Chapter 15 (the inference workload, latency-throughput trade-offs). The KV-crossover batch size $B^{\ast} = M_{\text{weights}} / C_{\text{kv}}$ is the first number to compute; it determines whether you are in the superlinear or sublinear throughput regime before you touch any configuration. You are choosing a parallelism strategy for a training run Chapter 12 derives the memory and communication cost of every combination: tensor-parallel, pipeline-parallel, data-parallel, ZeRO stages 1-3, and FSDP. The worked examples are parameterized so you can substitute your own model size and cluster topology. You are debugging a training run that is not hitting expected MFU Chapter 8 (arithmetic intensity and the roofline model) and Chapter 11 (collective communication bandwidth and all-reduce timing) are the diagnostic starting points. Low MFU is almost always communication-bound, memory-bound, or pipeline-bubble-bound; the chapters give you the arithmetic to distinguish the three cases. You are evaluating a KV cache optimization (GQA, MLA, paged attention) Chapter 16 derives the memory reduction and the resulting change in maximum batch size for each technique. The formulas let you estimate the throughput gain on your specific hardware and model before running any experiment. You are designing an evaluation harness for a fine-tuned model Chapters 23-25 cover evaluation design from first principles: metric selection, contamination detection, benchmark validity, and the gap between benchmark performance and deployment behavior. Chapter 25 covers domain-specific evaluation design (medical, legal, code) and deployment telemetry. You are deciding between SFT, DPO, and RLVR for a post-training objective Chapter 26 (SFT), Chapter 27 (DPO and RLHF), and Chapter 28 (GRPO and RLVR) each cover the method’s mechanics, its data requirements, its failure modes, and the scenarios where it is the right choice. Chapter 31 synthesizes the decision into an end-to-end alignment pipeline design.

Chapter Map by Engineering Decision

The Arithmetic Is the Point

Every number in production LLM engineering is derivable. When a colleague states that “a 70B model needs at least 4 H100s to serve,” the correct response is to ask whether that accounts for KV cache at the target batch size and context length, not to accept it as a fact. The derivation: $M_{\text{weights}} = 2 \times 70 \times 10^{9} \approx 140$ GB in bf16, spread across $\lceil 140 / 80 \rceil = 2$ H100s for weights alone, but a realistic batch size and context window can double that figure.

Chapter 15 and Appendix A derive these estimates in full, including the formulas for $B_{\max}$ (VRAM-limited batch size), $B^{\ast}$ (KV-crossover batch size), and $t_{\text{step}}$ (decode step latency as a function of batch size and model size). These are not rules of thumb; they are consequences of memory capacity and bandwidth, and they update correctly when you change the model, the hardware, or the context length.

Connecting the Book to the Engineering Literature

The book’s technical content is grounded in papers and engineering reports that have shaped the current state of production LLM systems. Where a chapter’s content derives from a specific paper, the source is cited inline. Appendix D provides an annotated reading list organized by topic, with each entry annotated to indicate which specific claims are worth understanding in depth rather than simply citing by name.

The FlashAttention algorithm (Chapters 9 and 10), the Chinchilla scaling law (Chapter 14), continuous batching (Chapter 18), and the GRPO training algorithm (Chapter 28) are examples where reading the original paper alongside the corresponding chapter will deepen your understanding of the design choices behind the method. The reading list indicates which papers repay that deeper reading and which are sufficient to know at the result level.

Staying Current

The LLM systems field moves fast. The core arithmetic and hardware physics in this book-memory bandwidth, arithmetic intensity, roofline analysis, parallelism trade-offs-are stable. Specific techniques (attention variants, serving schedulers, compression methods) continue to evolve. When a new technique is announced, the most reliable way to evaluate it is to ask: which constraint does it address, and what does it trade away? That question is the same one this book trains you to ask, and it does not go stale.