Chapter 15 The Inference Workload

15.1 Prefill vs. Generation

15.1.1 Compute-Bound Prefill

When an interviewer asks you to design a low-latency chat system and then asks “what determines your time-to-first-token?”, the answer bottoms out in prefill. This phase is what runs before the model emits a single output token: you feed the entire prompt through the transformer, computing all the intermediate activations and caching the key and value tensors for every layer. Getting this phase right is the central lever for TTFT, and understanding why it behaves differently from generation is a foundational piece of inference architecture knowledge that shows up at every seniority level.

Prefill is the phase in which all $S$ prompt tokens are processed in a single forward pass. Because all input tokens are known before execution begins, the attention computation can be fully parallelized across the sequence dimension-exactly as in training. The transformer reads an input matrix $X \in \mathbb{R}^{S \times d}$, passes it through $L$ layers of attention and MLP, and produces two outputs: the logit distribution over the vocabulary for the next token, and the KV cache that will be consumed by every subsequent generation step.

The key metric for understanding whether a computation is hardware-limited is arithmetic intensity: the ratio of floating-point operations to bytes transferred from high-bandwidth memory (HBM).

For an MLP layer during prefill, the dominant operation is two matrix multiplications (up-projection and down-projection). With batch size $B$ and $T = S$ tokens, the intensity evaluates to $B \cdot S$. For the attention layers, the QK$\top$V computation gives an intensity of roughly $S \cdot T / ( S + T )$, which simplifies to $S / 2$ when $T = S$ (as it is during prefill). Both expressions grow with sequence length. On an H100 the roofline crossover for a standard matrix multiply sits around 295 FLOPs/byte; prefill crosses that line comfortably once $S$ is even modestly long.

Prefill is compute-bound: the GPU’s arithmetic units are the bottleneck, not the memory bus, which means you are extracting close to peak FLOP/s from the hardware.

The reason prefill gets to be compute-bound is structural: every token’s query, key, and value projections involve the same weight matrices, and with $S$ tokens in flight simultaneously you are executing a true matrixmatrix multiply ($X \in \mathbb{R}^{S \times d}$, $W \in \mathbb{R}^{d \times f}$). Matrix-matrix multiplies have arithmetic intensity proportional to the matrix dimensions, so as $S$ grows you can fully hide the cost of loading weights from HBM behind arithmetic.

Compare this to the alternative-naive recomputation where you feed one token at a time, recomputing all prior context from scratch. The complexity per output token would be $O ( S^{2} )$ in FLOPs because you’d re-attend over the full prefix. The KV cache removes this redundancy: you pay $O ( S )$ FLOPs during prefill to populate it, and thereafter each generation step costs $O ( 1 )$ additional attention FLOPs per prior token. You are trading memory (storing the KV tensors) for computation (not recomputing them). At inference time, where you have no gradient bookkeeping, this trade is always worth it.

The practical consequence is that TTFT scales approximately linearly with $S$ under fixed batch size, and reducing batch size during prefill gives you better latency at the cost of throughput. Unlike generation, adding sequences to a prefill batch genuinely helps because the arithmetic intensity scales with $B \cdot S$.

A Worked Example

Consider Llama-2 13B running on a single H100 SXM, processing a prompt of $S = 2 , 048$ tokens with batch size $B = 1$. The key dimensions are $d = 5 , 120$, $f = 4 d = 20 , 480$, $L = 40$ layers, and $N = 40$ attention heads each of dimension $H = 128$. All tensors are in bf16 (2 bytes per element).

MLP arithmetic intensity. For one MLP layer:

Bytes transferred (weights + activations):

Intensity $\approx B \cdot S = 1 \times 2048 = 2048$ FLOPs/byte, well above the H100 roofline of 295. The MLP is compute-bound.

Attention arithmetic intensity. With $S = T = 2048$:

Also comfortably compute-bound.

Prefill latency estimate. Total FLOPs for a full forward pass are approximately $6 \cdot N_{\text{params}} \cdot S \approx 6 \times 13 \times 10^{9} \times 2048 \approx 1 . 6 \times 10^{14}$. At the H100’s theoretical bf16 throughput of $\sim 989$ TFLOP/s, compute time is roughly $\sim 160$ ms for this single long prompt. In practice TTFT for a 2K-token prompt on a 13B model lands in the 100-200 ms range on a single H100, consistent with being compute-bound at reasonable MFU.

KV cache footprint after prefill.

This 1.7 GB is allocated at the end of prefill and held until the request completes.

What You Actually Decide in Practice

• Prefill batch size governs TTFT. Reducing $B$ during prefill directly lowers TTFT because the compute time scales with $B \cdot S$. If your SLA is TTFT $< 200$ ms, you may need to cap concurrent prefill requests to 1-2, accepting lower GPU utilization during this phase.

• Long prompts become the bottleneck faster than you expect. At $S = 8 , 192$ tokens, prefill on a 13B model takes $\sim 640$ ms at $B = 1$. At $S = 32 , 768$ it is over 2.5 seconds. If your application has retrievalaugmented prompts or long system instructions, prefill latency will dominate your TTFT budget even before generation starts.

• Chunked prefill allows interleaving with generation. Some production serving systems split a long prompt into chunks (e.g., 512 tokens each) and interleave prefill chunks with ongoing generation steps for other requests. This smooths TTFT distribution at the cost of slightly higher per-request latency variance. Use it when your tail latency SLA is tight and you have concurrent requests.

• Model parallelism affects prefill differently than generation. Tensor parallelism across $T$ GPUs cuts prefill time by roughly $T \times$ because you are splitting a compute-bound operation. For generation, the same parallelism provides less benefit because the bottleneck shifts to memory bandwidth.

• Flash Attention changes the constant, not the asymptotic bound. Using FlashAttention-2 during prefill reduces HBM traffic by fusing the attention kernel and avoiding materializing the full $S \times S$ attention matrix. It improves prefill throughput by roughly 2-3$\times$ on long sequences, but the operation remains compute-bound; it does not change the fundamental scaling behavior.

How This Shows Up in Interviews

• “A user sends a 10,000-token system prompt. Your TTFT is 4 seconds. How do you fix it?” A strong answer covers: (1) chunked prefill to interleave with ongoing decode; (2) prompt caching-if multiple requests share the same system prompt prefix, compute the KV cache once and reuse it; (3) reducing tensor parallelism overhead by verifying all-reduce latency is not dominating; (4) checking GPU utilization to confirm you are actually compute-bound.

• “Why is prefill compute-bound but generation memory-bound?” Strong answer: during prefill you process $S$ tokens simultaneously, so weight reads are amortized over $S$ compute ops per weight element. During generation you process one token at a time, so each weight matrix is read from HBM to perform only a matrix-vector multiply-one compute op per weight element loaded.

• Senior follow-up: “Suppose you disaggregate prefill and decode onto separate GPU pools. What are the implications?” The senior answer covers: prefill-optimized hardware can favor high-FLOP-count GPUs (H100 SXM); decode-optimized hardware benefits more from high memory bandwidth per GPU (or more GPUs per parameter shard). Disaggregation also lets you scale each pool independently based on prompt-length distribution and output-length distribution, and eliminates the head-of-line blocking where a long prefill stalls ongoing decode steps.

Key Takeaways

• Prefill processes all prompt tokens in parallel, making it a matrix-matrix multiply with arithmetic intensity $\propto B \cdot S$-compute-bound on modern accelerators for any reasonable prompt length.

• The output of prefill is the populated KV cache; every generation step that follows reads from this cache, so prefill is a one-time amortized cost.

• TTFT scales linearly with prompt length and batch size. A 4$\times$ longer prompt means $\sim 4 \times$ longer TTFT.

• Prompt caching (reusing KV tensors across requests sharing a common prefix) can eliminate prefill cost entirely for repeated system prompts-a high-leverage optimization in production.

• Disaggregating prefill and decode onto separate GPU pools lets each be tuned for its regime: compute-intensive vs. memory-bandwidth-intensive.

15.1.2 Memory-Bound Generation and the KV Cache

Generation is the phase that breaks most engineers’ intuition the first time they benchmark an LLM in production. You have a massively parallel GPU with hundreds of tensor cores and 3.3 TB/s of memory bandwidth, and yet it is sitting at 5% compute utilization while producing tokens one by one at 80 milliseconds each. The culprit is arithmetic intensity: generating a single token is a sequence of matrix-vector multiplies, and matrixvector multiplies are almost always memory-bound on modern hardware. This bottleneck is the central fact around which every inference optimization-larger batches, grouped-query attention, quantization, speculative decoding-is organized. Understanding it at the arithmetic level is what separates a surface-level interview answer from a strong one.

After prefill, the model enters the autoregressive generation loop: at each step it takes one new token embedding, runs it through all $L$ transformer layers, and samples the next token. Crucially, only one new token is processed per step. The attention layers need to attend over all prior tokens, but the key and value projections for those prior tokens were already computed during prefill and cached in HBM-the KV cache.

The KV cache for a single sequence stores, for every layer, the key and value tensors for every token processed so far. Its total size is:

where $L$ is the number of layers, $N_{\text{kv}}$ is the number of key-value heads, $H$ is the head dimension, and $S_{\text{current}}$ grows by one at each step.

Now consider what the GPU must do to produce one token. For each MLP layer it reads a weight matrix $W \in \mathbb{R}^{d \times f}$ from HBM to multiply against a single activation vector $x \in \mathbb{R}^{d}$. The number of FLOPs is $2 d f$ and the bytes transferred are $2 d f$ (for the weight, in bf16) plus the tiny contribution from the vector itself. The arithmetic intensity is:

For attention, each generation step must also read back the full KV cache for all prior tokens ($S$ entries) for each of the $N_{\text{kv}}$ heads. Working through the FLOP and byte counts, the intensity simplifies to:

regardless of batch size. The H100’s roofline crossover is $\sim 295$ FLOPs/byte. An intensity of 1 means you are using less than 0.4% of the available compute; you are almost entirely bottlenecked by the speed of HBM reads.

Generation is memory-bound because producing each token reduces the transformer to a series of matrixvector multiplies and a KV cache read, both of which transfer far more bytes per FLOP than the hardware can hide.

The KV cache is still the right trade-off despite introducing the memory bottleneck. Without it, each generation step would recompute the key and value projections for all $S$ prior tokens from scratch, costing $O ( S )$ attention FLOPs per step. With the cache, attention per step costs $O ( S )$ bytes read (to load the cache) plus $O ( S )$ FLOPs for the dot products-but the prefill amortized the projection cost. You are trading computation for memory bandwidth, and for autoregressive generation this trade is always favorable: compute is the constrained resource during prefill, memory bandwidth is the constraint during generation, and the cache lets each resource be the binding constraint during its respective phase.

The MLP layers behave differently from attention in an important way. Their arithmetic intensity is $\approx B$ (the batch size), not 1. This means that by processing $B$ requests simultaneously, you amortize the cost of loading each weight matrix over $B$ activation vectors-you get $B$ useful FLOPs per byte loaded. At $B = 295$ on an H100 you cross the roofline and the MLP becomes compute-bound. Attention, however, gives you no such relief: because each sequence has its own KV cache, batching does not reduce the bytes loaded per sequence. The KV cache bytes and FLOPs both scale linearly with $B$, so the intensity stays pinned near 1 no matter how large the batch.

This asymmetry has a practical consequence that trips up engineers who only look at MLP utilization: even if your batch size is large enough to saturate the MLP layers, the attention layers remain memory-bound throughout generation. The attention layers are therefore the harder constraint to escape.

A Worked Example

Take Llama-2 13B on a single H100 SXM. The model has 13.0B parameters stored in bf16, occupying $\approx 26$ GB. The H100’s memory bandwidth is 3.35 TB/s.

Memory-limited latency per generation step. At $B = 1$, the dominant memory transaction per step is loading the model weights (26 GB) plus reading the KV cache for the current sequence length $S$. For $S = 1 , 000$ the KV cache is:

Total bytes to transfer: $26$ GB $+ 0 . 82$ GB $\approx 26 . 8$ GB. At 3.35 TB/s:

That gives a theoretical throughput of $1 / 0 . 008 = 125$ tokens/second at $B = 1$.

Scaling to larger batches. At $B = 16$, the KV cache grows to $16 \times 0 . 82 \approx 13$ GB, but the weights are still 26 GB and are loaded once per step and shared across all 16 sequences. Total bytes: $26 + 13 = 39$ GB. Latency: $\approx 12$ ms per step. But you generate 16 tokens in that step, so throughput is $16 / 0 . 012 \approx 1 , 330$ tokens/second-a $10 . 6 \times$ throughput gain from a $16 \times$ batch increase, reflecting diminishing returns as the KV cache begins to compete with the weights for bandwidth.

At $B = 256$ the KV cache alone reaches $256 \times 0 . 82 \approx 210$ GB, which overflows H100 VRAM (80 GB). The maximum batch size that fits in a single H100 is therefore roughly $B = 80$ GB total $- 26$ GB weights $= 54$ GB available for KV $\Rightarrow B \approx 66$ for $S = 1 , 000$.

The attention intensity, concretely. For a single generation step at $B = 64$, $S = 1000$:

Intensity $\approx 0 . 5$. Half a FLOP per byte: firmly in the memory-bandwidth regime, regardless of how many sequences are batched.

What You Actually Decide in Practice

• Batch size is your primary throughput knob, up to the memory wall. Every doubling of $B$ roughly doubles throughput until KV cache size starts to dominate the memory footprint. For a 13B model on an H100, this ceiling is around $B = 50$-$70$ for sequences of 1K tokens. Beyond that, you need to reduce sequence length, switch to GQA/MQA, or quantize the KV cache.

• KV cache quantization gives you headroom before adding GPUs. Quantizing KV cache entries from bf16 to int8 halves KV memory, doubling the effective maximum batch size. At $S = 1 , 000$ this moves the memory ceiling from $B \approx 65$ to $B \approx 130$ on a single H100 with 80 GB VRAM-a significant lever before you pay for more hardware.

• Grouped-query attention (GQA) is the most impactful architectural lever. Reducing from $N$ KV heads to $N / G$ KV heads shrinks the KV cache by a factor of $G$, with minimal accuracy impact at $G \leq 8$. This is why Llama-3 adopted GQA across all model sizes: the same weights serve more concurrent requests on the same hardware.

• At $B = 1$, generation is pure memory bandwidth. If you are running a low-concurrency service (a single-user IDE assistant, for example), your generation speed is almost exactly parameter_bytes / memory_bandwidth_bytes_per_second per token. You are not compute-limited; getting a faster GPU does not help unless its bandwidth is higher.

• Latency and throughput trade off directly through batch size. At $B = 1$ on Llama-2 13B / H100: latency $\approx 8$ ms/token, throughput $\approx 125$ tok/s. At $B = 32$: latency $\approx 10$ ms/token, throughput $\approx 3 , 200$ tok/s. You cannot get both low latency and high throughput from a single serving instance; use request routing or disaggregated serving if your SLAs require both.

How This Shows Up in Interviews

• “Why is LLM generation slow? What’s the bottleneck?” Strong answer: generation is memory-bound, not compute-bound. Each step is a series of matrix-vector multiplies (weight matrices times a single token embedding) and a KV cache read. The arithmetic intensity for both operations is $O ( 1 )$-$O ( B )$, far below the H100’s roofline of 295 FLOPs/byte. The GPU is bottlenecked by how fast it can stream model weights and KV cache from HBM, not by how fast its tensor cores can multiply.

• “How does increasing batch size help, and when does it stop helping?” Strong answer: larger $B$ amortizes the cost of loading weight matrices over more activation vectors, raising arithmetic intensity from $\sim 1$ toward the roofline for MLP layers. But attention layers give no benefit from batching because each sequence’s KV cache is independent-intensity stays near 1. The hard ceiling is GPU VRAM: once $B \times$ KV cache per sequence $+$ weights exceeds VRAM, the batch does not fit.

• Senior follow-up: “The attention arithmetic intensity is always $\approx 1$ no matter how large the batch. Is there any architectural change that can actually fix this?” Senior answer: yes, two approaches. First, reduce KV cache size so fewer bytes are transferred per step: GQA reduces KV heads, MLA projects KV to a lower-dimensional latent space, and local (sliding-window) attention discards cache entries beyond a window. Second, eliminate autoregression altogether for generation: diffusion-based language models generate all tokens in parallel and iterate, turning the problem into a compute-bound one and yielding 5-10$\times$ throughput improvements at the cost of more complex sampling dynamics.

Key Takeaways

• Generation reduces to matrix-vector multiplies and KV cache reads; both have arithmetic intensity $\leq B$, putting generation firmly in the memory-bandwidth-limited regime on all current hardware.

• Attention is worse than MLP during generation: its intensity is $\approx 1$ regardless of batch size because each sequence owns its own KV cache, eliminating the weight-amortization benefit of batching.

• The practical throughput ceiling for a single GPU is set by VRAM capacity: batch size $\leq ($VRAM $-$ weights$) /$KV cache per sequence.

• Every major inference optimization-GQA, MLA, KV quantization, speculative decoding-targets this memory bottleneck either by reducing bytes transferred or by restructuring the problem to increase FLOPs per byte.

• Latency and throughput are in direct tension: minimum-latency serving uses $B = 1$ and loads weights once per request; maximum-throughput serving packs the largest $B$ that fits in memory.

15.2 Latency, Throughput, and Cost

15.2.1 Time-To-First-Token and Inter-Token Latency

When you are designing a serving system in an interview, the first question is almost always “what are your latency targets?” The follow-up-the one that separates candidates who have shipped inference from those who have only read about it-is “which latency?” Time-To-First-Token (TTFT) and inter-token latency (ITL) are governed by fundamentally different physics, and conflating them leads to the wrong optimization strategy at every level of the stack. This distinction comes up at mid-level and above for any role touching production inference.

Time-To-First-Token is how long a user waits between submitting a request and seeing the first generated character. It is dominated by the prefill phase: encoding all $S$ prompt tokens through all $L$ transformer layers in one parallel forward pass. This is structurally identical to a training forward pass-all tokens are available simultaneously, so the computation parallelizes across the sequence. Prefill is compute-bound when $B \cdot T$ is large enough to saturate the accelerator.

Inter-token latency (also called decode latency or generation latency) is the time between consecutive output tokens after the first. Each step produces exactly one new token by loading the full model weights from HBM, performing a matrix-vector product against the current hidden state, and attending over the growing KV cache. The arithmetic intensity of this operation is $\approx 1$ FLOP/byte (one for attention, $B$ for MLP-see the previous subsection), so the step is memory-bandwidth-bound regardless of GPU generation.

The governing equation for per-token generation latency is:

HBM bandwidth BW where $P$ is the parameter count, $C_{\text{kv}}$ is the KV cache size per sequence (bytes), $B$ is the number of concurrent sequences, and BW is the peak memory bandwidth of the accelerator. The factor of 2 on $P$ reflects bf16 storage (2 bytes per parameter).

TTFT and ITL are governed by different physics-the former by compute throughput, the latter by memory bandwidth-and any design that treats them as a single “latency” number will optimize for the wrong thing.

Prefill is fast because the transformer can process all $S$ tokens in a single batched matrix multiply. The arithmetic intensity during prefill scales as $B \cdot S$: a batch of one request with a 4096-token prompt has intensity $\approx 4096$, comfortably above the H100’s roofline of $\approx 295$ FLOPs/byte. TTFT is therefore roughly proportional to $S$ and inversely proportional to sustained compute throughput-something you cannot improve without either reducing the prompt or adding accelerators.

Generation is slow because each autoregressive step is equivalent to a matrix-vector product: you are loading the full weight matrix $W \in \mathbb{R}^{D \times D}$ to compute $W x$ for a single vector $x \in \mathbb{R}^{D}$. The ratio of FLOPs to bytes is $2 D^{2} / ( 2 D^{2} ) = 1$, essentially 1 FLOP per byte transferred. Modern H100s achieve $\approx 989$ TFLOP/s and $\approx 3 . 35$ TB/s bandwidth, so the compute-to-bandwidth ratio is $989 / 3 . 35 \approx 295$. A batch size of 1 operates at intensity 1-$295 \times$ below what it would need to saturate compute. Without a KV cache the situation would be even worse: each generation step would re-encode the entire prefix at $O ( S )$ cost, giving $O ( S^{2} )$ total work. The KV cache trades memory for time, converting $O ( S^{2} )$ generation to $O ( S )$-a fundamental architectural choice that determines every other inference optimization.

A Worked Example

Consider Llama-2 13B on a single H100 SXM5 (80 GB HBM, 3.35 TB/s bandwidth). Model configuration: $D = 5120$, $L = 40$, $N_{\text{heads}} = 40$, $N_{\text{kv}} = 40$, head dimension $H = 128$, sequence length $S = 1024$.

Model weight memory:

KV cache per sequence at $S = 1024$:

(The first factor of 2 is for keys and values; the last 2 is for bf16.)

Per-token ITL at $B = 1$:

TTFT at $S = 1024$, $B = 1$ (compute-bound prefill):

A user sending a 1024-token prompt waits $\approx 27$ ms before the first token appears, then receives tokens every $\approx 8$ ms. Doubling the prompt to 2048 tokens roughly doubles TTFT to $\approx 54$ ms. ITL stays constant because it depends on model size, not prompt length. At 100k-token prompts-common in document-analysis workloads-TTFT climbs to $\approx 2 . 7$ s even before a single output token is generated.

What You Actually Decide in Practice

• Set separate SLOs for TTFT and ITL. A latency budget of “$< 200$ ms” is ambiguous. Reasonable starting points for interactive chat: TTFT $< 500$ ms, ITL $< 50$ ms. Batch pipelines can relax TTFT to seconds while holding ITL tight for tail latency.

• Quantize weights to cut ITL directly. Switching from bf16 to INT8 halves $M_{\text{weights}}$, cutting ITL roughly in half at no change to model architecture. INT4 cuts it to $\approx 25 \%$ of bf16. For a 13B model: INT8 brings ITL from $8$ ms to $\approx 4$ ms. This is the cheapest ITL reduction available before touching architecture.

• TTFT grows linearly with prompt length; prune prompts aggressively. Every additional 1000 prompt tokens adds $\approx 2 \times 13$B $\times 1000 / 989$T $\approx 26$ ms to TTFT on a 13B model. Summarization or truncation strategies that halve average prompt length halve average TTFT.

• Chunked prefill trades TTFT for system-level ITL. When a long prompt arrives, splitting it into fixedsize chunks processed across multiple steps allows the GPU to interleave prefill with generation steps from other in-flight requests. The chunked request’s TTFT increases, but queued requests no longer stall behind a multi-second prefill job. Use this when p99 ITL matters more than p50 TTFT.

• Multi-GPU tensor parallelism reduces TTFT, not ITL, proportionally. Splitting the model across $K$ GPUs splits the prefill FLOPs by $K$, cutting TTFT roughly $K$-fold. ITL also improves because $M_{\text{weights}}$ per GPU drops, but you now pay inter-GPU communication latency on every generation step-a non-trivial overhead on large $K$. For latency-sensitive serving, prefer $K = 2$ or $K = 4$; beyond that, communication dominates.

How This Shows Up in Interviews

• “What’s the difference between TTFT and inter-token latency, and which matters more for a streaming chat product?” Strong answer: TTFT is prefill time (compute-bound, scales with prompt length and model size); ITL is per-token generation time (memory-bandwidth-bound, scales with model weight bytes and batch KV cache size). For streaming chat, ITL dominates user experience: a 200 ms TTFT is invisible, but 100 ms ITL makes text feel sluggish. Optimize ITL via quantization and KV cache reduction; optimize TTFT via multi-GPU prefill or prompt pruning.

• “You’re serving a 70B model and a user sends a 50k-token prompt. Walk me through what happens to TTFT.” Strong answer: TTFT scales as $2 P \cdot S /$compute. At 50k tokens on a single H100: $2 \times 70$B $\times 50000 / 989$T $\approx 7$ s. This is unacceptable for interactive use. Solutions: distribute prefill across multiple GPUs (pipeline or tensor parallel), use chunked prefill with a short-context draft first, or compress the prompt before sending.

• Senior follow-up: “How does chunked prefill interact with the KV cache, and what does it do to tail latency for other requests?” Senior answer: Chunked prefill processes the prompt in $\lceil S / C \rceil$ chunks of size $C$ tokens each. Between chunks, the scheduler can insert generation steps for other active requests. The KV cache for the chunked request grows incrementally-each chunk appends to it-so memory usage is identical to non-chunked prefill at completion. The benefit is that long-prompt requests no longer monopolize the GPU for seconds at a time, dramatically reducing p99 ITL for the rest of the request queue. The cost is a $O ( S^{2} / C )$ increase in total prefill FLOPs due to partial attention recomputation across chunk boundaries-tunable by choice of $C$.

Key Takeaways

• TTFT is dominated by prefill: compute-bound, scales linearly with prompt length $S$ and model size $P$.

• ITL is dominated by memory bandwidth: ITL $\approx ( 2 P + B \cdot C_{\text{kv}} ) /$BW; for a 13B bf16 model at $B = 1$ on H100, expect $\approx 8$ ms/token.

• The two metrics are decoupled: halving model weights halves ITL but does not change TTFT; halving prompt length halves TTFT but does not change ITL.

• Quantization (bf16 $\rightarrow$ INT8) is the highest-leverage, lowest-risk way to reduce ITL; it applies immediately to any trained model.

• Always set separate SLOs; a single “latency” target will cause you to over-optimize one metric at the expense of the other.

15.2.2 Throughput Trade-Offs Under Batching

Every LLM serving team hits the same wall within weeks of launch: per-request latency looks fine at one query per second, but the GPU sits at $30 \%$ utilization and cost-per-token is an order of magnitude higher than projected. The answer is batching-processing multiple requests simultaneously to amortize model weight loads across more useful work. But batching has its own ceiling, and the math behind that ceiling explains both why your throughput stalls and what architectural choices (GQA, quantization, replica scaling) give you the biggest gains. This trade-off is a core topic at senior and staff levels wherever inference cost matters.

Throughput is the total number of tokens generated per second across all users, in contrast to per-user latency which measures a single request’s experience. During the memory-bound generation phase, the system performs one decode step per token, and the cost of that step is:

BW where $B$ is the number of concurrent sequences, $C_{\text{kv}}$ is the KV cache footprint per sequence, and BW is HBM bandwidth. Each step produces $B$ tokens, so throughput is:

throughput$( B ) = \frac{B}{\text{step\_time} ( B )} = \frac{B \cdot \text{BW}}{M_{\text{weights}} + B \cdot C_{\text{kv}}}$

As $B \rightarrow \infty$ (ignoring memory capacity), throughput approaches BW$/ C_{\text{kv}}$-a hard ceiling set entirely by KV cache size per sequence. At the other extreme, $B = 1$ gives throughput $=$ BW$/ ( M_{\text{weights}} + C_{\text{kv}} )$, which is dominated by weight memory. The crossover batch size $B^{\ast}$ at which KV cache equals model weight memory is:

Below $B^{\ast}$, increasing $B$ improves throughput rapidly because the fixed weight cost is amortized over more tokens. Above $B^{\ast}$, KV cache dominates and throughput gains diminish quickly.

The KV-crossover batch size $B^{\ast}$ is the most important number in your serving capacity model-it tells you exactly how much batching buys you and where to stop.

At $B = 1$, loading 26 GB of weights produces one token: you get 1 token per weight-load. At $B = 32$, the same weight load is shared across 32 decode steps happening in parallel, amortizing the dominant cost over 32 useful outputs. This is the same principle as batched matrix multiplication: the $B$ dimension in $X W$ ($X \in \mathbb{R}^{B \times D}$, $W \in \mathbb{R}^{D \times F}$ ) lets each byte of $W$ contribute to $B$ outputs rather than one.

Contrast this with the obvious alternative of simply running $B$ independent single-batch requests serially: you load weights $B$ times, burn $B \times$ the memory bandwidth, and produce the same $B$ tokens. Batching collapses $B$ weight loads into one.

The cost is per-request latency: if $B$ requests are batched together, the slowest one determines when all of them complete their current step. This creates the fundamental throughput-latency trade-off: more batching $\Rightarrow$ higher throughput, higher tail latency. Replica parallelism-launching $M$ independent copies of the model, each handling its own requests at a lower batch size-sidesteps this trade-off entirely, delivering $M \times$ throughput at unchanged latency. Replicas require no inter-GPU communication and are the first scaling lever you should reach for.

A Worked Example

Llama-2 13B on a single H100 (80 GB HBM, 3.35 TB/s bandwidth, $S = 1024$):

• $M_{\text{weights}} = 26$ GB (bf16)

• $C_{\text{kv}} = 0 . 84$ GB per sequence (from Section 15.2.1)

KV-crossover batch size:

Above $B = 31$, KV cache dominates.

Throughput at various batch sizes:

At $B = 32 \approx B^{\ast}$, throughput is $\approx 16 \times$ higher than $B = 1$ but step time has only doubled. At $B = 64$ (above $B^{\ast}$), the incremental throughput gain from doubling $B$ from 32 is only $1 . 3 \times$-diminishing returns have clearly set in. $B = 128$ does not fit in 80 GB HBM without reducing KV cache size via GQA or quantization.

Effect of GQA (Llama-3 style, $N_{\text{kv}} = 8$ instead of 40):

GQA shifts the crossover point from 31 to 155, meaning you can fit $B = 128$ in 26 + 128 $\times$ 0.168 $\approx$ 47 GB-comfortably inside the 80 GB limit-at nearly 5$\times$ higher throughput than the original $B = 64$ cap.

What You Actually Decide in Practice

• Compute $B^{\ast}$ before setting any serving configuration. $B^{\ast} = M_{\text{weights}} / C_{\text{kv}}$. Throughput gains below $B^{\ast}$ are superlinear; above $B^{\ast}$ they are sublinear. For Llama-2 13B at $S = 1024$, $B^{\ast} \approx 31$. For the same model at $S = 4096$, $C_{\text{kv}}$ quadruples and $B^{\ast}$ drops to $\approx 8$-meaning longer contexts hurt your batching efficiency dramatically.

• Use replica parallelism before increasing batch size. $M$ replicas give exactly $M \times$ throughput with identical per-request latency and zero communication overhead. If your cost budget allows three H100s and latency is acceptable at $B = 1$, three replicas at $B = 1$ beats one replica at $B = 3$ in both latency and simplicity. Increase $B$ only when you need to serve traffic that exceeds what replicas alone can handle.

• Use continuous (dynamic) batching. Static batch sizes waste cycles: if one of $B$ requests finishes early, its slot sits idle until all $B$ requests complete. Continuous batching allows the scheduler to insert a new request into a finished slot at the next decode step without waiting for a full batch to drain. This is table stakes in any production serving stack (vLLM, TGI, TensorRT-LLM all implement it).

• GQA is the highest-leverage architectural change for throughput. Reducing $N_{\text{kv}}$ from 40 to 8 (a 5$\times$ reduction) raises $B^{\ast}$ by 5$\times$ and expands max viable $B$ by 5$\times$, yielding $2$-$4 \times$ real-world throughput improvement without changing model quality meaningfully. If you are choosing a base model for a high-traffic product, GQA support is a first-order selection criterion.

• Cost per token scales inversely with throughput. On an H100 at $4/hr and throughput of 2,689 tok/s ($B = 64$), cost is $4 / 3600 / 2689 \approx \$ 0 . 0004$/1k tokens. At $B = 1$ (125 tok/s), cost is $\approx \$ 0 . 009$/1k tokens- $22 \times$ higher. Maximizing batch size within the HBM limit is by far the largest cost lever before model changes.

How This Shows Up in Interviews

• “How do you maximize throughput for a high-traffic LLM serving endpoint?” Strong answer: Profile memory first. Compute $B^{\ast} = M_{\text{weights}} / C_{\text{kv}}$ and the maximum $B$ that fits in HBM. Use continuous batching to approach that maximum. Apply GQA or INT8 quantization to raise both $B^{\ast}$ and $\text{max} B$. Scale replicas horizontally for the remaining multiplier. Every step here has a concrete number attached; never hand-wave about “utilizing the GPU better.”

• “A user reports high tail latency at peak traffic. Walk me through root-causing it.” Strong answer: The most common cause at scale is batch size growth: as QPS rises, the scheduler packs more requests per step to maintain throughput, increasing per-step latency and pushing tail requests to wait longer. Confirm via step-time histogram. Fix options in priority order: (1) add replicas to spread load, reducing effective $B$ per replica; (2) implement preemption to cap $\text{max} B$; (3) implement priority queues so latency-sensitive requests get smaller batches.

• Senior follow-up: “Walk through the full memory budget for serving Llama-2 70B at $B = 32$, $S = 2048$, on two H100s with tensor parallelism.” Senior answer: Model weights (bf16): $2 \times 70 \times 10^{9} = 140$ GB, split across two GPUs $= 70$ GB each. KV cache: $2 \times 80 \times 64 \times 128 \times 2048 \times 2 \times 32 =$ roughly $\approx 68$ GB total (not sharded in basic tensor parallelism). Total per GPU: $70 + 34 = 104$ GB-exceeds the 80 GB limit. Solutions: shard KV cache across GPUs alongside weights (sequence parallelism), reduce $B$ to $\approx 14$, or use GQA to shrink $C_{\text{kv}}$. This is the kind of capacity calculation you should be able to do on a whiteboard in two minutes.

Key Takeaways

• Throughput scales as $B \cdot$ BW$/ ( M_{\text{weights}} + B \cdot C_{\text{kv}} )$; gains are superlinear below the KV-crossover $B^{\ast} = M_{\text{weights}} / C_{\text{kv}}$ and sublinear above it.

• For Llama-2 13B (bf16) on H100 at $S = 1024$: $B^{\ast} \approx 31$, $\text{max} B \approx 64$ (memory-limited), peak throughput $\approx 2 , 700$ tok/s-$22 \times$ higher than $B = 1$.

• Replica parallelism scales throughput linearly at zero latency cost and zero communication overhead; exhaust replicas before increasing batch size.

• GQA reduces $C_{\text{kv}}$ proportionally to the head reduction ratio, raising $B^{\ast}$ and $\text{max} B$ by the same factor.

• Cost per token is inversely proportional to throughput; maximizing $B$ within HBM limits is the largest cost lever available without architectural changes.