Chapter 29 Building a Production LLM Serving Stack

29.1 Reference Architecture

29.1.1 Gateway, Router, and Autoscaler

When an interviewer asks you to “design a production LLM serving system,” what they want to see is not a single GPU running a model loop. They want the full stack: the layer that faces the internet, the layer that decides which replica handles each request, and the layer that expands or contracts the fleet as traffic shifts. This question appears at every senior systems or ML-infra interview at frontier labs, and candidates who conflate “inference efficiency” with “serving architecture” rarely get the offer. The two concerns interact, but they live at different abstraction levels, and you need to be fluent in both.

A production LLM serving stack decomposes into three collaborating planes. The gateway is the publicfacing API endpoint: it authenticates callers, enforces rate limits, validates payloads, and streams tokens back to clients using server-sent events or WebSocket connections. The router sits behind the gateway and is responsible for request dispatch: given an incoming prompt, it selects a backend replica (or a prefill/decode worker pair in disaggregated setups) and hands the request off. The autoscaler monitors aggregate queue depth, GPU utilization, and SLO attainment and adjusts replica count accordingly.

The three components share a common data plane: a token stream flowing from the inference workers back through the router and out the gateway, and a control plane: health signals, load metrics, and scheduling decisions flowing in the other direction. In practice, the gateway is often a thin reverse proxy (Nginx, Envoy, or a cloud load balancer), the router is a custom service aware of model-specific state (particularly KV cache residency), and the autoscaler is a policy engine running against a metrics store.

Two metrics govern every SLO conversation at this layer. Time-to-first-token (TTFT) measures the wall-clock gap between request arrival and the moment the client receives the first generated token. TTFT is dominated by prefill latency because the model must encode the entire prompt before it can emit anything. Inter-token latency (ITL), sometimes called decode latency, measures the gap between successive tokens after the first. ITL is dominated by memory bandwidth because generation is memory-bound, not compute-bound. Throughputtokens generated per second across all concurrent users-is the aggregate metric the operator cares about when planning capacity.

Every architectural decision in the serving stack is ultimately a negotiation between TTFT, ITL, and throughput, and you cannot improve all three simultaneously.

Separating gateway, router, and autoscaler is not bureaucratic over-engineering. Each component has a radically different scaling profile. The gateway needs horizontal scale proportional to request rate; it is stateless and trivially replicated. The router needs to carry soft state: knowledge of which replicas have which KV blocks cached, what their current queue depths are, and what prefix hashes look like. Merging router logic into the gateway produces a monolith that cannot scale the two concerns independently. The autoscaler, meanwhile, is a control loop with a multi-minute decision horizon; it has no business sitting on the hot path of individual requests.

The deeper argument for this decomposition comes from the physics of GPU inference. Prefill (encoding the prompt) is compute-bound: given a long enough sequence the arithmetic intensity exceeds the H100’s compute-to-bandwidth ratio of roughly 295, so the GPU stays busy. Decode (generating tokens one at a time) is memory-bound: with batch size 1 the arithmetic intensity collapses to approximately 1 FLOP per byte, nearly 300$\times$ below what the hardware can sustain. This asymmetry means the optimal allocation of GPU time for prefill and decode is very different, and a router that is unaware of which stage a request is in will make poor dispatch decisions. The alternative-a single monolithic worker that handles both phases-wastes GPU-hours on the slower decode phase and creates queuing pressure for new arrivals whose TTFT suffers while they wait behind long decode tails.

A Worked Example

Consider a 13B-parameter model served on H100 SXM5 GPUs (80 GB HBM, 3.35 TB/s memory bandwidth, 989 TFLOPS BF16). Model parameters in BF16 consume $13 \times 10^{9} \times 2 = 26$ GB. Each H100 can hold the model and leave roughly 50 GB for KV cache.

The KV cache per sequence is:

where $N_{k v}$ is the number of key/value heads (for a vanilla 13B with 40 heads, $N_{k v} = 40$), $H = 128$ (head dimension), $L = 40$ layers, $T$ is the sequence length (1024 tokens), and the factor of 2 at the front accounts for both key and value, and 2 bytes for BF16. This yields $2 \times 40 \times 128 \times 40 \times 1024 \times 2 \approx 838$ MB per sequence. With 50 GB free, a single GPU can hold at most around $50 , 000 / 838 \approx 59$ sequences concurrently.

At batch size 1, decode latency is:

latency $=$ parameters bytes $+$ KV bytes$= 26 , 000 + 838$ MB$\approx 8$ ms per token

Throughput is $1000 / 8 = 125$ tokens/second for one sequence. At batch size 32, the memory footprint becomes $26 , 000 + 32 \times 838 \approx 52 , 800$ MB and latency is $52 , 800 / 3 , 350 , 000 \approx 15 . 8$ ms, but throughput is $32 / 0 . 0158 \approx 2 , 025$ tokens/second-a $16 \times$ throughput gain at the cost of $2 \times$ latency. This arithmetic is what the autoscaler must internalize: adding replicas is the right move when queue depth is climbing and per-request latency is still within SLO; increasing per-replica batch size is the right move when replicas have headroom but throughput targets are not met.

What You Actually Decide in Practice

• TTFT SLO drives prefill isolation. If your TTFT SLO is under 500 ms and prompt lengths routinely exceed 4k tokens, you will almost certainly need to disaggregate prefill onto dedicated high-compute nodes and decode onto high-memory-bandwidth nodes. A single fleet trying to do both will violate TTFT under bursty load because long decodes hold the GPU while new prefills queue.

• Router state granularity matters. A stateless round-robin router wastes KV cache: the same system prompt gets re-encoded on every replica. A prefix-hash-aware router routes requests with shared prefixes to the replica that already holds those KV blocks, cutting TTFT for chat history, RAG context, and system prompts to near zero for cache hits. The cost is router complexity and the need for a fast prefix-hash store (typically a Redis or in-memory hash table).

• Autoscaler lag is your biggest latency risk. GPU instances take 2-5 minutes to start and load a 13B model. If you scale reactively, you will absorb a traffic spike entirely as latency degradation. Scale proactively: watch the ratio of queue depth to mean decode time per step. If that ratio exceeds your replica’s max concurrent sequence count, trigger scale-out before queues overflow.

• Continuous batching, not static batching. Static batching waits until a fixed batch fills before dispatching. With variable-length completions this wastes GPU cycles waiting for stragglers. Continuous batching (as implemented in vLLM) inserts new requests into a running decode step as soon as a sequence slot frees, keeping the batch dense at all times. This is non-negotiable for production.

• Model replicas scale throughput linearly; tensor parallelism does not. Launching $M$ independent copies of the model across $M$ GPUs multiplies throughput by $M$ at constant latency. Sharding one model across $M$ GPUs via tensor parallelism reduces per-request latency but introduces all-reduce communication overhead and does not scale throughput unless you also add replicas. Use tensor parallelism only when the model does not fit on a single GPU; prefer replicas otherwise.

How This Shows Up in Interviews

• “Walk me through what happens when a user hits your LLM API endpoint.” Strong answers name each layer: TLS termination and auth at the gateway, routing policy at the router, prefill on a dedicated or shared worker, incremental token streaming back through the gateway, and where TTFT and ITL are measured. Weak answers start at the GPU and work inward.

• “How would you reduce TTFT for a system serving 10k concurrent users?” A strong answer distinguishes the two root causes: long prompts slow prefill (fix with disaggregated prefill workers or prompt compression), while large queue depths slow dispatch (fix with more replicas or better scheduling priority for new arrivals). A weak answer says “use a faster GPU.”

• “How does your router avoid recomputing the same system prompt on every request?” The answer is prefix caching: hash the system prompt tokens, store the associated KV blocks on each replica, and route requests to the replica holding the match. State that cache eviction follows an LRU policy and that the KV store must be block-granular (as in PagedAttention) to avoid pinning entire sequences.

• Senior follow-up: “Your autoscaler is scaling in correctly but P99 TTFT is still spiking every 30 minutes. What’s happening?” The senior answer diagnoses a garbage-collection analogue: KV cache eviction storms. As sequences complete, their blocks are freed; if the scheduler allows too many long-context requests to pile up simultaneously, memory pressure forces the eviction of partially-warm cache blocks mid-batch, causing re-prefill for evicted sequences and a TTFT spike. The fix is admission control at the router: cap the number of sequences admitted based on remaining KV capacity, not just compute capacity.

Key Takeaways

• The production serving stack has three separable planes-gateway, router, autoscaler-each with a distinct scaling profile and state requirement.

• TTFT is dominated by prefill (compute-bound); ITL is dominated by decode (memory-bound); these two phases have different hardware optima, which motivates disaggregation.

• At batch size 1 on an H100, a 13B model generates tokens roughly every 8 ms; batching to 32 costs $2 \times$ latency but delivers $16 \times$ throughput-the autoscaler must understand this curve.

• Prefix-hash-aware routing is the single highest-leverage router optimization: it converts redundant prefill work into cache hits with no accuracy cost.

• Continuous batching and admission control (capping inflight sequences by KV capacity) are non-negotiable for production; static batching and reactive scaling will both fail under real traffic patterns.

29.1.2 KV-Cache-Aware Scheduling

A scheduler that treats GPU memory as a black box will, under real traffic, either starve low-latency requests waiting behind long-context ones or thrash the KV cache into a pattern of constant eviction and re-prefill. Both failure modes show up as P99 TTFT spikes that no amount of extra hardware fixes. KV-cache-aware scheduling is the discipline of making dispatch, admission, and eviction decisions that account for exactly how many physical KV blocks each in-flight sequence holds and how much capacity each candidate replica has left. It is a systems question that appears at senior and staff interviews whenever the conversation moves from “how does inference work” to “how do you run it at scale.”

The KV cache is the dominant memory consumer during decode. For each in-flight sequence the system must store one key tensor and one value tensor per layer, per head, per token seen so far. In BF16 the per-token cost per layer is:

where $N_{k v}$ is the number of key/value heads, $H$ is the head dimension, and the leading factor of 2 counts both K and V. Summed over $L$ layers, a sequence of length $T$ occupies $2 \times N_{k v} \times H \times L \times T \times 2$ bytes. Because this scales linearly with $T$, a single 32k-token sequence can consume as much KV memory as dozens of 1k-token sessions.

The core insight of KV-cache-aware scheduling is that admitting a new request is an irreversible memory commitment until the sequence finishes or is explicitly preempted. A scheduler that does not track block occupancy will over-admit, forcing the system to either stall (waiting for a slot to free) or evict a partially-decoded sequence and re-prefill it from scratch, wasting the compute already spent. The PagedAttention abstraction (used in vLLM and its successors) makes the accounting tractable: KV memory is partitioned into fixed-size blocks (typically 16 tokens wide), and the scheduler tracks a free-block count rather than raw bytes. A request is admitted only when enough free blocks exist to hold its full worst-case KV footprint, or when a preemption policy frees sufficient blocks first.

Prefix caching extends this further. Many requests share a common prefix-a system prompt, a RAG context, or a few-shot template. If the KV blocks for that prefix are already resident on a replica, the prefill cost for the shared portion is zero. The scheduler therefore maintains a prefix hash table: a mapping from a rolling hash of the token sequence to the physical block addresses that hold the corresponding KV vectors. When a new request arrives, the scheduler computes its prefix hash and routes it to the replica with the longest matching cached prefix, turning prefill into a cache hit.

Admission control, block-level eviction policy, and prefix-aware routing are three levers that together determine whether the KV cache operates near its theoretical efficiency or wastes the majority of its capacity on redundant computation.

The alternative-FIFO dispatch without block accounting-fails in two characteristic ways. First, a long-context request admitted at the front of the queue holds tens of thousands of blocks for minutes; shorter requests queue behind it even though the GPU has cycles to spare on a smaller batch. Second, when total block occupancy approaches the physical limit, a naively greedy system must either halt new admissions (starving throughput) or evict and re-prefill the longest resident sequence (wasting $O ( T )$ compute that has already been paid).

Block-level accounting solves the first problem by decoupling scheduling from request order: the dispatcher picks from the ready queue whichever request best fills available block capacity without over-committing. It solves the second by making eviction explicit and policy-driven. The standard policy is a modified LRU over logical sequences: when blocks must be freed, the scheduler selects the sequence with the lowest priority score (often a combination of elapsed wait time, sequence length, and whether the sequence has a live client connection). In practice, background batch jobs are prime eviction candidates; interactive sessions are protected.

Prefix caching compounds the gains. A system-prompt block that is reused across 1000 sessions is computed exactly once. Without prefix routing, each session independently prefills that prompt, turning a constant cost into a cost proportional to request rate. At 10k requests per minute with a 2k-token system prompt at 8 ms per prefill, unrouted sessions waste roughly $10 , 000 \times 8 / 60 \approx 1 , 333$ GPU-ms per minute on a single replicacompute that otherwise could have served decode steps.

A Worked Example

Take a Llama-3-8B model (GQA with $N_{k v} = 8$ key/value heads, $H = 128$, $L = 32$ layers) running on an H100 with 80 GB HBM. Parameters in BF16 consume $8 \times 10^{9} \times 2 = 16$ GB. KV cache budget is $80 - 16 - 2 = 62$ GB (reserving 2 GB for activations and workspace).

KV bytes per token:

With a block size of 16 tokens, each block holds $16 \times 128$ KB $= 2$ MB of KV data. The 62 GB budget supports:$62 \times 1024$ MB

A request with a 4k-token prompt plus a 1k-token max completion requires $\lceil 5000 / 16 \rceil = 313$ blocks at peak. The scheduler can safely admit $\lfloor 31 , 744 / 313 \rfloor = 101$ such requests before the H100 is fully committed. Admitting request 102 without eviction forces an out-of-memory stall.

Now add prefix caching. Suppose all requests share a 1k-token system prompt ($\lceil 1000 / 16 \rceil = 63$ blocks). After the first request warms those blocks, every subsequent request needs only $\lceil 4000 / 16 \rceil = 250$ blocks for its unique portion. Effective capacity rises to $\lfloor ( 31 , 744 - 63 ) / 250 \rfloor = 126$ concurrent requests-a 25% capacity gain with no hardware change.

What You Actually Decide in Practice

• Block size is a latency-fragmentation tradeoff. Smaller blocks (8 tokens) reduce internal fragmentation but increase block-table overhead and TLB pressure. Larger blocks (32 tokens) cut overhead but waste memory when sequences end mid-block. Most production systems settle at 16 tokens; drop to 8 only for extreme short-context workloads where fragmentation dominates.

• Set admission headroom, not a hard limit. Admitting up to the last block is dangerous: decode steps allocate one new block per step per sequence, so a fully-packed batch will OOM mid-step. Reserve a safety margin of 5-10% of total blocks for in-flight growth. The scheduler should refuse new admissions when free blocks fall below this watermark, not when they reach zero.

• Preemption policy must match your SLO tier. For interactive users, never preempt; instead, queue the incoming request. For batch jobs, preempt aggressively using swap-to-CPU (slower but preserves progress) or full eviction (faster but wastes prefill). Implement priority tiers at the request level and wire them into the eviction comparator.

• Prefix hash granularity determines cache hit rate. Hashing at the full-prompt level misses partial prefix matches. Hash at block granularity: compute a rolling hash after each block boundary and store a hit for every prefix length that is a multiple of the block size. This maximises reuse when system prompts vary by a few appended tokens, which is common in multi-turn conversations.

• Disaggregated prefill changes the scheduler contract. When prefill workers are separate from decode workers, the scheduler must coordinate a handoff: the prefill worker streams KV blocks for the prompt into shared memory or over NVLink, and the decode worker picks up from there. The decode-side scheduler must reserve the necessary blocks before the handoff begins, or the arriving blocks have nowhere to land. Reserve-then-transfer, not transfer-then-find-space.

How This Shows Up in Interviews

• “How does vLLM prevent out-of-memory errors at high load?” Strong answers explain PagedAttention block accounting: the scheduler maintains a free-block counter and refuses admission when the counter minus the safety watermark is less than the worst-case block requirement of the incoming request. Eviction policy (LRU over logical sequences, swap vs. discard) should be named. Weak answers say “it manages memory better.”

• “Two replicas are available. How do you route an incoming request?” The strong answer: compute the prefix hash of the request’s prompt, check each replica’s prefix hash table for the longest matching cached prefix, and route to the winner. If neither has a match, route to the replica with the most free blocks. Fall back to round-robin only if block counts are equal and no prefix match exists. Naming the cost of a wrong routing decision-a full re-prefill-makes the answer concrete.

• “Your P99 TTFT spikes every few minutes on an otherwise healthy cluster. What’s your diagnosis?” The candidate should describe a KV eviction storm: the decode batch fills available blocks, admission control kicks in, new requests queue, and when a batch of long sessions finally completes it releases a burst of blocks that triggers a flood of newly admitted requests whose prefills saturate the prefill stage simultaneously. The fix is admission rate limiting: spread re-admissions over multiple scheduling steps rather than unblocking the entire queue at once.

• Senior follow-up: “How would you implement prefix caching across replicas, not just within one?” The senior answer introduces a distributed prefix store-a fast key/value service (Redis Cluster or a custom shard-by-hash service) mapping prefix hashes to the replica ID that holds the warm blocks. On cache hit across replicas, the router either redirects the request or triggers a block migration via RDMA. Block migration is worthwhile only when the shared prefix exceeds a threshold (roughly 512 tokens) because migration latency amortizes over the TTFT saving. Below that threshold, local re-prefill is faster.

Key Takeaways

• KV memory is block-granular and grows linearly with sequence length; the scheduler must track free blocks explicitly and refuse admission before-not after-the last block is consumed.

• Prefix caching converts repeated prefill work into cache hits; a Llama-3-8B H100 instance gains roughly 25% effective concurrency when a 1k-token system prompt is shared across all sessions.

• Eviction policy must respect SLO tiers: interactive sessions are never preempted; batch jobs are the first candidates for swap-to-CPU or full eviction.

• Block-granular prefix hashing maximises cache hit rate for multi-turn conversations where prompts differ only in appended tokens.

• In disaggregated prefill/decode deployments, block reservation must happen on the decode worker before KV blocks are transferred from the prefill worker; the reverse order causes mid-transfer OOM and wastes the prefill compute.

29.2 Capacity Planning

29.2.1 Estimating Tokens-Per-Second per GPU

Capacity planning for a production serving system begins with a single derivable number: how many output tokens per second can one GPU produce for your model and workload? Every downstream decision-fleet size, cost projection, SLA commitment-anchors to this estimate. In interviews at frontier labs, the question arrives within the first ten minutes: “We need to serve a 70B model at 1,000 requests per second with p95 latency under 100 ms. How many H100s?” Candidates who reach for benchmark sheets fail; candidates who derive the answer from the physics of memory bandwidth on the whiteboard pass. This is a senior-level question because it requires internalizing why generation is fundamentally different from training.

Autoregressive generation is memory-bandwidth-bound. Every time the model generates a single token, the GPU must sweep all model weights from high-bandwidth memory (HBM) into registers, execute the forward pass, and write effectively nothing back-a deeply asymmetric, read-dominated workload. For a 13B-parameter model in BF16, that sweep transfers $13 \times 10^{9} \times 2 = 26$ GB per token, per batch step, regardless of batch size.

To generate a token for a batch of $B$ concurrent sequences, the GPU must also read each sequence’s KV cache. In BF16, the KV cache for one sequence of length $S$ (tokens already generated or prefilled) is:

where $L$ is the number of transformer layers, $K$ is the number of KV heads, $H$ is the head dimension, and the factor of 4 combines 2 bytes per element (BF16) with 2 tensors (key and value). The total bytes transferred per generation step for a batch of size $B$ is:

where $P$ is the parameter count. Given hardware memory bandwidth $W$ (bytes per second), the per-step latency is $M ( B , S ) / W$, and batch throughput in tokens per second is:

TPS$( B , S ) =_{\text{seq}}$ This formula has two regimes. When $B \cdot C_{\text{seq}} \ll 2 P$-small batch, weights dominate-TPS scales linearly with $B$. When $B \cdot C_{\text{seq}} \gg 2 P$-large batch, KV cache dominates-TPS saturates at the ceiling $W / C_{\text{seq}} ( S )$, independent of any further increase in $B$.

To maximize GPU utilization during generation, the primary lever is batch size $B$; every generated token forces a full sweep of model weights and all per-sequence KV caches from HBM.

The root cause is arithmetic intensity: the ratio of floating-point operations to bytes transferred. For an MLP projection with weight matrix of shape $D \times F$, processing a batch of $B$ tokens requires $2 B D F$ FLOPs and transfers roughly $2 D F + 2 B D + 2 B F \approx 2 D F$ bytes (when $D$ and $F$ dwarf $B$). The intensity simplifies to approximately $B$ FLOP/byte. An H100 SXM5 has a ridge point of $989$ TFLOP/s$\div 3 . 35$ TB/s $\approx 295$ FLOP/byte. You need $B \geq 295$ to be compute-bound on MLP layers alone.

Attention layers are worse. Each sequence maintains its own KV cache, so unlike MLP weights-shared across all batch members-the KV data scales with $B$. The arithmetic intensity of one attention step during generation is approximately $S T / ( S + T ) \approx 1$ when $T = 1$ (one new token per step), entirely independent of batch size. No amount of batching recovers efficiency in the attention path during decode.

The practical consequence is stark. During prefill, both the prompt length $T = S$ and the batch $B$ can be large, driving arithmetic intensity above the ridge point; the GPU is compute-bound and operates near peak FLOPs. During generation, $T = 1$, intensity collapses to roughly $B$ for MLP and $1$ for attention, and the GPU is almost always memory-bandwidth-bound. This asymmetry is why building a high-throughput decoder is fundamentally a batching problem, not a kernel-optimization problem.

A Worked Example

Take Llama-2-13B on a single H100 SXM5: $P = 13 \times 10^{9}$, $L = 40$ layers, $K = 40$ KV heads, $H = 128$ head dimension, $W = 3 . 35$ TB/s, 80 GB HBM. In BF16:

Free HBM after loading weights: $80 - 26 = 54$ GB. Maximum concurrent sequences at $S = 1 , 000$ tokens: $\lfloor 54 / 0 . 78 \rfloor = 69$.

Latency at $B = 1$: $( 26 + 0 . 78 )$ GB$/ 3 , 350$ GB/s $\approx 8 . 0$ ms. At $B = 16$: $( 26 + 12 . 5 ) / 3 , 350 \approx 11 . 5$ ms. At $B = 64$ (near the memory ceiling): $( 26 + 49 . 9 ) / 3 , 350 \approx 22 . 7$ ms. Throughput is $B /$latency.

Raising the batch from 1 to 64 multiplies TPS by $22 . 6 \times$ at the cost of $2 . 8 \times$ higher per-request latency-a favorable trade for batch workloads, a painful one for interactive users who measure inter-token latency personally. The theoretical TPS ceiling as $B \rightarrow \infty$ is $W / C_{\text{seq}} = 3 , 350 / 0 . 78 \approx 4 , 295$ TPS; in practice, the memory limit of 69 concurrent sequences bounds you to about 2,900 TPS at best.

To serve 500 requests per second at an average output length of 200 tokens, you need $500 \times 200 = 100 , 000$ output TPS. At $\approx 2 , 900$ TPS per H100, the baseline fleet is $\lceil 100 , 000 / 2 , 900 \rceil = 35$ GPUs, plus 20-30% headroom for traffic bursts and prefill capacity.

What You Actually Decide in Practice

• Target memory bandwidth, not peak FLOPs, when evaluating hardware. Upgrading from an A100 (2.0 TB/s) to an H100 (3.35 TB/s) delivers a $1 . 7 \times$ TPS increase for memory-bound generation. Switching from BF16 to INT4 weights halves bytes transferred per step, roughly doubling throughput-a larger gain than a full GPU generation upgrade. Profile your actual arithmetic intensity before committing to hardware SKUs.

• Use continuous batching to keep $B$ near $B_{\max}$. Static batching waits for a full batch to assemble before dispatching, leaving GPU memory idle while requests queue. Continuous batching inserts new requests into an active decode step the moment a sequence slot frees, maintaining dense GPU occupancy at all times. Under bursty traffic, continuous batching routinely delivers $2 \times$ the throughput of static batching at the same hardware cost.

• Disaggregate prefill and decode under sustained load. A 4K-token prefill for one user stalls 64 decoding sequences for its full duration-roughly $4 , 000 \times 8$ ms $\approx 32$ s of blocked inter-token latency at $B = 1$; less at larger batches, but still a significant TTFT spike. At scale, route prefill requests to dedicated compute-heavy nodes and stream KV caches to decode-only nodes. This disaggregation is increasingly standard at 100B-parameter deployments.

• Replicate before sharding for models under 80B. A 13B model fits on a single H100 with 54 GB remaining for KV caches. Running 8 independent replicas delivers $8 \times$ throughput at zero inter-GPU communication cost. An 8-way tensor-parallel shard reduces per-request latency but adds all-reduce overhead on every layer and does not improve aggregate throughput unless you also replicate the sharded group. Use tensor parallelism only when the model does not fit on a single GPU.

• Budget 25-30% of HBM as system overhead. KV cache fragmentation, CUDA context, flash-attention internal buffers, and OS overhead reliably consume this margin. An H100 with 80 GB is safely usable to about 56-60 GB for model and KV cache combined. Planning at 95% occupancy causes OOM crashes during traffic spikes and leaves no room for model updates without a full service restart.

How This Shows Up in Interviews

• “How would you estimate how many H100s you need to serve a 70B model at 1,000 RPS?” A strong answer derives TPS from first principles: compute $2 P = 140$ GB (requires tensor parallelism across two H100s per replica), compute $C_{\text{seq}}$ at expected context length, derive $B_{\max}$ per replica pair, then compute TPS per replica pair and divide into the required output TPS. State assumptions on average output length and context length explicitly before doing arithmetic.

• “Why is autoregressive generation memory-limited rather than compute-limited?” A strong answer names the mechanism: at $T = 1$ per step, MLP arithmetic intensity reduces to $\approx B$ FLOP/byte and attention intensity to $\approx 1$ FLOP/byte; you need $B \geq 295$ on an H100 to reach the MLP compute ridge point. At $B = 1$ you are $295 \times$ below compute saturation. Every additional user in the batch recovers efficiency linearly until the KV cache fills HBM and throughput saturates.

• Senior follow-up: “Your batch is already at memory capacity. What are your levers to push TPS higher?” The strong senior answer identifies two compounding levers: (1) reduce $C_{\text{seq}}$ via GQA or MLA-fewer KV heads means larger $B_{\max}$ at the same HBM budget, so the same hardware now serves proportionally more users; (2) reduce $2 P$ via quantization-INT8 weights halve the weight-sweep cost, shifting the crossover point where KV cache dominates, which allows doubling the batch size at the same per-step latency. Both are applied multiplicatively.

Key Takeaways

• Generation is memory-bandwidth-bound: TPS$( B , S ) = B \cdot W / ( 2 P + B \cdot C_{\text{seq}} )$, where $W$ is HBM bandwidth and $C_{\text{seq}} = 4 L K H S$ is KV cache bytes per sequence.

• TPS grows linearly with $B$ when weights dominate ($B \cdot C_{\text{seq}} \ll 2 P$) and saturates at $W / C_{\text{seq}}$ when KV cache dominates.

• Maximum batch size is $B_{\max} = \lfloor ( M_{\text{GPU}} - 2 P ) / C_{\text{seq}} \rfloor$; size fleets around this constraint, not peak FLOP specs.

• Continuous batching maintains near-$B_{\max}$ occupancy; static batching wastes capacity by idling while a batch assembles.

• For sub-80B models, prefer independent replicas over tensor parallelism: $N$ replicas deliver $N \times$ throughput at zero communication overhead.

29.2.2 Budgeting for Long-Context Workloads

Teams that extend their serving context window from 4K to 32K tokens routinely discover their GPU fleet is suddenly undersized by an order of magnitude. The capacity estimate that held last quarter is now wrong by a factor of eight or more. Context length is not a dial you can turn freely after sizing your fleet; it restructures the entire HBM budget, shrinks the maximum batch size, and degrades throughput in a way that compounds rather than adds. Interviewers at infrastructure-focused teams probe this directly: “If we double context length, how does our serving capacity change?” The answer is never “it roughly halves.” Understanding precisely why-and by how much-separates a strong senior answer from a weak one.

The KV cache grows linearly with sequence length $S$, but its effect on serving capacity is superlinear because a longer cache simultaneously reduces the maximum concurrent batch size and increases the bytes transferred per generation step. The KV cache per sequence is:

With GPU HBM capacity $M_{\text{GPU}}$ and model parameters consuming $2 P$ bytes, the maximum batch size is:

At maximum batch occupancy, nearly all HBM is consumed by weights and KV caches. The total memory swept per generation step is approximately $M_{\text{GPU}}$, so the per-step latency is roughly constant:

The batch throughput is therefore:

When a GPU is packed to its HBM limit, generation throughput scales inversely with context length: doubling $S$ halves TPS, and the GPU stays fully occupied the entire time.

This $1 / S$ relationship holds when the memory constraint is binding. At shorter contexts where you have fewer concurrent users than $B_{\max}$-throughput is limited by request arrival rate, not hardware-the scaling breaks; context length barely matters because unused KV slots cost nothing.

The intuition is that HBM is a fixed pool shared between model weights and KV caches. At full batch occupancy, almost all of it contains KV data. Doubling $S$ doubles $C_{\text{seq}}$, which forces you to halve $B_{\max}$ to fit in memory. The per-step latency stays approximately the same (you are still sweeping $\approx M_{\text{GPU}}$ bytes per step), but you produce half as many tokens per step. Throughput halves.

This stands in sharp contrast to prefill, where longer sequences actively help: a larger prompt length $T = S$ during encoding raises arithmetic intensity above the ridge point, keeping the GPU compute-bound. The degradation from long context is entirely a decode-phase problem.

Techniques that reduce $C_{\text{seq}} ( S )$ attack this problem at the source. GQA with $g$-fold head reduction changes $K \rightarrow K / g$, shrinking $C_{\text{seq}}$ by $g$ and growing $B_{\max}$ by $g$-a proportional TPS improvement at every context length. MLA (Multi-head Latent Attention) projects KV tensors into a low-rank bottleneck of dimension $c \ll K \cdot H$, reducing the per-token KV footprint to $4 L c$ bytes; DeepSeek-V2 compressed from $K \cdot H = 16 , 384$ to $c = 512$, a $32 \times$ reduction. Both techniques are most impactful at long context because that is when the KV cache, not the weights, is the binding constraint on batch size.

A Worked Example

Use Llama-2-13B on a single H100 ($M_{\text{GPU}} = 80$ GB, $W = 3 . 35$ TB/s), with $2 P = 26$ GB leaving $54$ GB for KV caches. The per-token KV cost per sequence is:

TPS is computed as $B_{\max} \cdot W / ( 2 P + B_{\max} \cdot C_{\text{seq}} )$. Going from $S = 1 , 000$ to $S = 32 , 000$ (a $32 \times$ context increase) reduces TPS from 2,900 to 88-a $33 \times$ degradation, matching the $1 / S$ prediction closely.

Fleet sizing consequences are severe. If you need 100,000 output TPS (500 RPS at 200 output tokens per response):

• At $S = 1 , 000$: $\lceil 100 , 000 / 2 , 900 \rceil = 35$ H100s

• At $S = 16 , 000$: $\lceil 100 , 000 / 176 \rceil = 569$ H100s

A $16 \times$ context extension drives a $16 \times$ GPU cost increase-before accounting for inter-node networking and orchestration overhead. Context length is a business decision, not just an engineering dial.

What You Actually Decide in Practice

• Segment users into context-length tiers with separate serving pools. A single 128K-token request occupies roughly 160 sequence slots worth of KV cache at $S = 1 , 000$ per slot. Mixing workloads in one pool lets long-context requests starve short-context users of GPU memory without any apparent failure mode-just silent throughput collapse. Maintain separate fleets for short ($< 4$K), medium (4K-32K), and long ($>$32K) tiers, and price each tier to reflect its actual resource multiplier.

• Apply GQA before scaling the fleet for long-context workloads. Going from $K = N = 40$ (full multi-head) to $K = 8$ (8 KV heads) cuts $C_{\text{seq}}$ by $5 \times$, multiplying $B_{\max}$ and TPS by $5 \times$ at every context length. For Llama-2-13B at $S = 16 , 000$, this shifts $B_{\max}$ from 4 to 20 and TPS from 176 to $\approx 880$-recovering most of the throughput lost to long context before buying a single additional GPU.

• Enable prefix caching for shared long contexts. If $B = 100$ concurrent users all share a 4K-token system prompt, without prefix caching each sequence consumes $C_{\text{seq}} ( 5 , 000 ) \approx 3 . 9$ GB of KV cache (5K total tokens), giving $B_{\max} = \lfloor 54 / 3 . 9 \rfloor = 13$. With prefix caching, the shared 4K prefix KV (3.1 GB) is stored once; each sequence only needs $C_{\text{seq}} ( 1 , 000 ) = 0 . 78$ GB for its unique tokens, giving $B_{\max} = \lfloor ( 54 - 3 . 1 ) / 0 . 78 \rfloor = 65$-a $5 \times$ improvement in concurrent capacity with no hardware change.

• Use KV cache eviction rate as your capacity signal. Set an alert threshold at $5 \%$ eviction under normal load. Higher eviction means KV blocks for in-flight sequences are being expelled before those sequences complete, forcing expensive KV recomputation and spiking TTFT. If eviction climbs above $10 \%$ during sustained load, you are underprovisioned for the current context-length distribution; add capacity or tighten admission control before the queues overflow.

• At $C_{\text{seq}} ( S ) > M_{\text{GPU}} - 2 P$, switch to multi-GPU KV sharding. For Llama-2-13B at $S = 69 , 000$ tokens, a single sequence’s KV cache ($\approx 56$ GB) exceeds all available HBM after loading weights. Ring attention shards the KV cache across $N$ GPUs, restoring useful $B_{\max}$ values at the cost of all-gather communication at each layer-typically 10-20% latency overhead at NVLink bandwidth. This communication penalty is acceptable compared to the alternative of serving zero concurrent requests.

How This Shows Up in Interviews

• “A customer wants to extend context from 8K to 128K tokens. What happens to serving costs?” A strong answer derives $C_{\text{seq}} ( 128 , 000 )$ for the specific model: for Llama-2-13B, $4 \times 40 \times 40 \times 128 \times 128 , 000 \approx 100$ GB-already larger than the entire H100 HBM. Even with model weights excluded, one sequence at 128K context cannot fit on a single H100. The answer must name ring attention or sequence parallelism as requirements, then estimate GPU count from there. A weak answer says “you need more GPUs” without quantifying.

• “How does GQA improve long-context throughput?” A strong answer connects the mechanism to the formula: GQA reduces $K$ by factor $g$, shrinking $C_{\text{seq}}$ by $g$, multiplying $B_{\max}$ by $g$, and therefore multiplying TPS by $\approx g$ at any context length. The effect is largest at long context because that is when KV cache is the binding constraint rather than request arrival rate. State a concrete number: going from 40 to 8 KV heads on a 13B model at 16K context multiplies TPS by $5 \times$.

• Senior follow-up: “Your prefix cache has an 80% hit rate on a 4K shared system prompt for requests with 1K unique context. Quantify the throughput impact.” The senior answer reasons from effective KV footprint. Hit sequences carry $0 . 78$ GB (unique 1K KV only); miss sequences carry $3 . 9$ GB (full 5K KV). Average per-sequence KV is $0 . 8 \times 0 . 78 + 0 . 2 \times 3 . 9 = 1 . 40$ GB, plus $3 . 1$ GB for the cached prefix pinned in GPU memory. Effective $B_{\max} = \lfloor ( 54 - 3 . 1 ) / 1 . 40 \rfloor = 36$ versus $\lfloor 54 / 3 . 9 \rfloor = 13$ without any caching-a $2 . 8 \times$ throughput improvement.

Key Takeaways

• When a GPU is packed to its HBM limit, TPS $\propto 1 / S$: doubling context length halves generation throughput while the GPU stays fully busy.

• $B_{\max} ( S ) = \lfloor ( M_{\text{GPU}} - 2 P ) / C_{\text{seq}} ( S ) \rfloor$ is the hard capacity limit; it shrinks proportionally as context grows.

• GQA and MLA reduce $C_{\text{seq}}$ directly; a $g$-fold reduction in KV heads multiplies $B_{\max}$ and TPS by $g$ at every context length.

• Prefix caching eliminates repeated KV computation for shared context; at high hit rates it is equivalent to adding GPU replicas at near-zero hardware cost.

• When $C_{\text{seq}} ( S ) > M_{\text{GPU}} - 2 P$, a single sequence does not fit on one GPU; ring attention or KV offloading becomes mandatory, each carrying a 10-20% latency penalty.