Chapter 16 Reducing the KV Cache

16.1 Attention Variants for Cheaper Inference

16.1.1 Multi-Query and Grouped-Query Attention (GQA)

When an interviewer asks you to design a serving stack for a 70B model and your napkin math shows that the KV cache will overflow VRAM at any reasonable batch size, grouped-query attention is the first architectural lever to reach for. GQA is the change that moved from research curiosity to default in every serious open-weight model-Llama 3, Mistral, Qwen, Falcon-within about a year of its introduction, and interviewers at inferenceheavy teams will expect you to know it cold and be able to quantify its impact.

Standard multi-head attention (MHA) computes $N$ query heads, $N$ key heads, and $N$ value heads per token per layer. The KV cache therefore stores $2 \times N \times H$ values per token per layer, where $H$ is the head dimension. Because every sequence in a batch owns its own independent KV cache, this cost scales linearly with batch size, sequence length, and layer count, and it is the primary reason generation remains memory-bound regardless of how many queries you batch together.

Multi-query attention (MQA) pushed this to the extreme: share a single key head and a single value head across all $N$ query heads. The KV cache shrinks by a factor of $N$, but in practice MQA degrades model quality noticeably, especially at larger scales-the single shared KV head is not expressive enough.

Grouped-query attention (GQA) occupies the Pareto-optimal middle ground. You partition the $N$ query heads into $G$ groups of $N / G$ heads each, and each group shares one key head and one value head. Defining $K = N / G$ as the number of KV heads:

KV cache size per token per layer $= 2 \times K \times H \times$ bytes per element

Setting $K = N$ recovers MHA; setting $K = 1$ recovers MQA. In practice $G = 8$ (i.e., $K = N / 8$) preserves accuracy nearly identically to MHA while cutting the KV cache by $8 \times$.

GQA is the lowest-cost architectural change that shrinks the KV cache without meaningful accuracy loss, enabling either lower latency at the same batch size or a larger batch at the same latency.

The accuracy robustness of GQA comes from what the key and value projections actually learn. Within a group of $N / G$ query heads, the queries are already diverse-each attends to different positions and features. The key and value vectors, by contrast, encode the information content at each position, which is less sensitive to which specific query is asking. Sharing KV within a group turns out to be a mild constraint; the queries still specialize while the KV tensors carry shared context.

The throughput gain is two-fold. First, the direct reduction: $K$ KV heads instead of $N$ means $N / K = G$ times fewer bytes transferred from HBM per generation step, directly proportional to speed-up since generation is memory-bandwidth-bound. Second, an indirect gain: the freed VRAM can accommodate a larger batch size. Before GQA, a 13B model at $S = 1 , 000$ and $B = 64$ required over 50 GB just for the KV cache on a standard MHA model; with GQA at $G = 8$ that drops to 6 GB, leaving room to push $B$ well above 200 on a single 80 GB H100.

The comparison point is MQA. MQA also reduces memory but pays a quality tax that compounds at larger model scales. GQA effectively shows that the quality-memory trade-off curve is non-linear: you can recover almost all of MHA’s quality with only $K = N / 8$ KV heads rather than $K = N$, a point on the curve that gives you most of the memory benefit for almost none of the accuracy cost.

A Worked Example

Consider Llama-2 70B: $N = 64$ query heads, $H = 128$, $L = 80$ layers, bf16 (2 bytes per element), with batch size $B = 32$ and sequence length $S = 2 , 048$.

MHA KV cache. Per token per layer: $2 \times 64 \times 128 \times 2 = 32 , 768$ bytes. Total:

This overflows a single 80 GB H100 even before counting the 140 GB of model weights. You cannot run this batch on one GPU.

GQA with $G = 8$ ($K = 8$). Per token per layer: $2 \times 8 \times 128 \times 2 = 4 , 096$ bytes. Total KV cache:

Combined with weights: $140 + 21 . 5 = 161 . 5$ GB-fits comfortably on two H100s, and a single H100 can handle $B \leq 10$ at this sequence length.

Throughput estimate at $B = 32$ with GQA, two H100s. Total bytes transferred per generation step: $161 . 5$ GB. Aggregate bandwidth: $2 \times 3 . 35 = 6 . 7$ TB/s. Step latency: $161 . 5 / 6700 \approx 24$ ms. Throughput: $32 / 0 . 024 \approx 1 , 333$ tokens/second. Without GQA, this batch simply does not fit.

Accuracy check. Empirically, GQA at $G = 8$ recovers $\geq 99 \%$ of MHA perplexity across standard benchmarks for models at 7B-70B scale. The quality cost is negligible for downstream tasks.

What You Actually Decide in Practice

• Default to $G = 8$ for any model $\geq 7$B. This is the empirically validated sweet spot. Below 7B you may still use $G = 4$ or even MQA without noticeable downstream degradation because the models are less sensitive to head expressiveness.

• GQA changes your VRAM ceiling, not just your speed. The more important effect is often that GQA makes a batch size possible that was previously impossible, not that it speeds up an already-fitting batch. Always recompute your max batch size after adopting GQA-the number often increases by $4$-$8 \times$.

• Weight sharing means the projection weights are shared too. In CLA the weights are explicitly shared across layers; in GQA, the same $W_{K}$ and $W_{V}$ matrices produce the keys and values used by multiple query heads within a group. If you are loading a GQA checkpoint, the key-value projection matrices are smaller than query projection matrices by a factor of $G$.

• GQA is free at training time, not just inference time. Training with GQA is slightly faster than MHA because you compute fewer KV projections. Models trained from scratch with GQA converge to the same loss as MHA models; you do not need to fine-tune or distill a pre-trained MHA model into GQA.

• For $G > 16$, check accuracy on your target task. The gap between GQA and MHA is small for $G \leq 8$, but it widens for aggressive settings. For retrieval-heavy workloads (RAG, long-context question answering), the KV diversity may matter more than for generation tasks-run an eval before committing to extreme compression ratios.

How This Shows Up in Interviews

• “Your KV cache is overflowing VRAM at the batch size you need. How do you fix it architecturally?” Strong answer: the primary lever is GQA-reduce the number of KV heads from $N$ to $N / G$ with $G = 8$ as the default, shrinking the KV cache by $8 \times$ with negligible accuracy impact. Secondary levers include KV quantization (int8 halves the cache further) and sliding-window attention on non-critical layers.

• “What is the difference between MQA, GQA, and MHA? When would you use each?” Strong answer: MHA uses $N$ KV heads (maximum expressiveness, maximum cache), MQA uses 1 KV head (minimum cache, measurable quality loss at scale), GQA uses $K = N / G$ KV heads sharing one KV pair per group of $G$ queries (best accuracy-memory trade-off, used in all modern production models). MQA is now obsolete for large models; GQA is the default; MHA survives only in legacy architectures or small models where memory is not the bottleneck.

• Senior follow-up: “You’re training a new 70B model from scratch. A colleague proposes using MHA because it’s simpler and GQA is a “trick.” How do you respond, and what training experiments would settle the debate?” Senior answer: GQA is not a post-hoc trick-it is a design decision made at model initialization. Train two identical models at 7B scale (a proxy) with $G = 8$ GQA and MHA respectively, evaluate on a downstream task suite and measure KV cache size at target serving batch size. If GQA shows $< 1 \%$ degradation, adopt it for the 70B run. The inference cost reduction justifies the negligible quality delta in almost every production setting.

Key Takeaways

• GQA reduces the number of KV heads from $N$ to $K = N / G$, shrinking the KV cache by a factor of $G$ with near-zero accuracy loss at $G \leq 8$.

• The cache reduction is doubly beneficial: fewer bytes per step means faster generation, and freed VRAM enables larger batch sizes that further improve throughput.

• For Llama-2 70B with $G = 8$, the KV cache at $B = 32$, $S = 2 , 048$ drops from $\sim 171$ GB (MHA, impossible on one or two H100s) to $\sim 22$ GB (GQA, easily fits alongside model weights).

• GQA is applied at training time, not as a post-hoc compression step; production models from Llama 3 onward are trained with GQA by default.

• MQA ($G = N$) is the degenerate case-maximum memory savings but unacceptable accuracy loss at scale; GQA is the right balance.

16.1.2 Multi-Head Latent Attention (MLA)

GQA attacks the KV cache along one axis: reduce the number of heads. Multi-Head Latent Attention (MLA), introduced in DeepSeek V2, attacks a different axis: project the key and value tensors into a shared low-dimensional latent before storing them, then reconstruct the full-rank KV at query time. The compression ratios MLA achieves-up to $31 \times$ in DeepSeek’s configuration-dwarf what GQA alone can deliver, and MLA is the design behind some of the most inference-efficient frontier models available. Senior interviews at labs building or deploying frontier models will expect you to understand the mechanism and its implications.

In standard MHA, each token at each layer stores $2 \times N \times H$ values in the KV cache: $N$ key vectors and $N$ value vectors each of dimension $H$. GQA reduces $N$ to $K = N / G$ but leaves each head’s dimensionality $H$ untouched. MLA takes the complementary approach: keep all $N$ heads but compress their joint representation into a single latent vector of dimension $C \ll N \times H$.

Concretely, MLA introduces a down-projection matrix $W^{D K V} \in \mathbb{R}^{d \times C}$ and an up-projection matrix $W^{U K} \in \mathbb{R}^{C \times ( N \times H )}$ for keys (and an analogous pair for values). The cached quantity per token per layer is the compressed latent:

At query time, the full key matrix is reconstructed on the fly:

The cache therefore stores $C$ values per token per layer instead of $2 \times N \times H$. In DeepSeek V2, $N = 128$, $H = 128$, so the uncompressed KV size is $2 \times 128 \times 128 = 32 , 768$ values per token. With $C = 512$, the cache stores only 512 values-a $64 \times$ compression on the stored representation, though in practice a small number of extra dimensions must be retained for positional encoding (see below), bringing effective compression to roughly $31 \times$.

The one structural complication is RoPE compatibility. Standard RoPE applies a position-dependent rotation to the key and query vectors before the attention dot product. With MLA, the keys are reconstructed from the latent $c_{t}$, which is position-agnostic. The solution DeepSeek adopted is to cache an additional small set of decoupled RoPE keys alongside the latent, keeping positional information accessible without expanding the main cache significantly.

MLA stores a low-rank compressed latent of the KV representation rather than the full KV tensors, achieving compression ratios far beyond what head-count reduction alone can provide, at the cost of up-projection compute at every query step.

The key intuition is that across all $N$ heads in a layer, the key and value projections of a single token are not $2 \times N \times H$ independent numbers-they share substantial structure because they all derive from the same token embedding $x_{t} \in \mathbb{R}^{d}$. A rank-$C$ projection can capture most of this structure when $C$ is chosen appropriately relative to the intrinsic dimensionality of the token-space representation.

The trade-off MLA makes is compute for memory. Every query step now incurs an additional matrix multiply to up-project the cached latent back to full KV-$C \times ( N \times H )$ FLOPs per head per token in the KV cache. Since generation is memory-bandwidth-bound, spending a modest amount of additional compute is a favorable trade if it dramatically reduces the bytes transferred. Whether the up-projection cost is worth it depends on the compression ratio $C / ( 2 N H )$: at DeepSeek’s $31 \times$ compression, the saved bandwidth far outweighs the extra multiply.

Compared to GQA, MLA offers much higher compression but is substantially harder to implement. GQA requires only reshaping the KV tensors during the attention kernel; MLA requires custom kernel support for the up-projection step inside the attention computation, and the RoPE decoupling adds further complexity. This is why GQA has seen broad adoption across many model families while MLA has so far stayed primarily within the DeepSeek ecosystem.

A Worked Example

DeepSeek V2 uses 60 transformer layers, $N = 128$ query heads, $H = 128$, and $C = 512$ for the compressed KV latent, plus $d_{r o p e} = 64$ extra dimensions for decoupled RoPE keys, giving a total cached size of $512 + 64 = 576$ values per token per layer.

KV cache comparison at $B = 1$, $S = 128 , 000$ (128K context).

Full MHA: $2 \times 128 \times 128 \times 128 , 000 \times 60 \times 2$ bytes $= 503$ GB. Infeasible on any single machine for interactive serving.

MLA: $576 \times 128 , 000 \times 60 \times 2$ bytes $\approx 8 . 8$ GB. Fits on a single 80 GB H100 alongside model weights.

Compression ratio: $32 , 768 / 576 \approx 56 . 9 \times$ in theory, $\approx 31 \times$ accounting for the RoPE overhead and practical implementation details.

Up-projection cost per generation step. At each step, for each of the $S = 128 , 000$ cached tokens, you perform one up-projection: $C \rightarrow N \times H$ for keys and an analogous one for values. FLOPs per layer per step:

At H100’s 989 TFLOP/s: roughly 2 ms per layer per step. Summed over 60 layers this is significant-roughly 120 ms of extra FLOPs per token at full 128K context. In practice this is fused with the attention kernel and partially hidden behind memory latency, so the effective overhead is lower. The bandwidth savings (from reading 576 values instead of 32,768 per cached token) dominate at long contexts.

Breakeven context length. MLA only wins when the bandwidth savings exceed the extra compute cost. For DeepSeek’s numbers, this breakeven is roughly $S \approx 4 , 000$ tokens-well within the range of practical long-context serving. Below 4K context, the overhead of up-projection matters more and MLA may not outperform GQA.

What You Actually Decide in Practice

• MLA is the right choice when targeting very long contexts ($S \geq 8 , 000$) and bandwidth is the binding constraint. The bandwidth reduction scales with $S$ while the up-projection cost scales identically-so MLA’s advantage grows with context length. For short-context workloads, GQA is simpler and competitive.

• The up-projection kernels must be custom. Standard attention kernels (FlashAttention, cuDNN) do not natively support MLA’s expand-then-attend pattern. Deploying MLA requires either a custom CUDA kernel or a serving framework that has integrated MLA support. Factor this into your build vs. buy decision when selecting a serving stack.

• Latent dimension $C$ is a hyperparameter with a clear sensitivity. In DeepSeek’s ablations, reducing $C$ below 256 caused measurable perplexity increases; $C = 512$ maintained accuracy near the MHA baseline. Use $C \approx d / 8$ as a starting point, where $d$ is the model dimension.

• RoPE decoupling adds a fixed overhead. The extra $d_{r o p e} = 64$ dimensions stored per token per layer are unavoidable if you want full positional expressiveness. Plan for this in your KV cache sizing-at long contexts it is a small fraction of the latent size but worth accounting for in exact memory calculations.

• GQA and MLA are orthogonal in principle but rarely combined in practice. GQA reduces head count; MLA reduces per-head dimensionality. A model could use both simultaneously, but the engineering complexity is additive. In current deployments, teams pick one: GQA for simplicity and broad ecosystem support, MLA for maximum compression at long context.

How This Shows Up in Interviews

• “Explain how MLA differs from GQA as a strategy for reducing the KV cache.” Strong answer: GQA reduces the number of KV head pairs from $N$ to $K = N / G$, cutting cache size by $G \times$. MLA keeps all $N$ heads but projects their joint representation into a $C$-dimensional latent before caching, cutting size by $2 N H / C \approx 31 \times$ in DeepSeek’s configuration. The trade-off MLA makes is additional up-projection compute at query time in exchange for dramatically lower HBM traffic.

• “What is the RoPE compatibility problem with MLA and how is it solved?” Strong answer: standard RoPE applies position-dependent rotations to key vectors before caching. In MLA the cache stores position-agnostic latents, so RoPE cannot be pre-applied. DeepSeek’s solution is to cache a small set of decoupled RoPE keys alongside the latent ($d_{r o p e} = 64$ extra dimensions), preserving positional information without expanding the main cache significantly.

• Senior follow-up: “You are designing a new model targeting 128K context serving at scale. Walk through the KV cache architecture decision from first principles.” Senior answer: start with the memory budget (e.g., one 80 GB H100, 30 GB for weights, 50 GB for KV at peak batch). Compute the per-token KV footprint for MHA, GQA, and MLA at your target $N$ and $H$. For 128K context even GQA may overflow; MLA at $C = 512$ shrinks this to sub-10 GB territory. Weigh the custom kernel cost against the serving economics-if you are running hundreds of instances, the engineering investment in MLA pays back quickly. Also factor in that MLA’s advantage grows with $S$, so it is the correct architectural choice for a long-context-first design.

Key Takeaways

• MLA caches a compressed low-rank latent $c_{t} \in \mathbb{R}^{C}$ per token per layer rather than full KV tensors, achieving $\sim 31 \times$ compression vs. MHA at DeepSeek V2’s settings ($C = 512$, $N = 128$, $H = 128$).

• The trade-off is compute: each query step must up-project cached latents back to full KV, adding FLOPs that are favorable when bandwidth savings dominate-typically at context lengths $\geq 4 , 000$ tokens.

• RoPE compatibility requires caching a small set of decoupled positional keys alongside the latent; this adds a fixed overhead of $d_{r o p e}$ values per token per layer.

• MLA is architecturally orthogonal to GQA (head count vs. head dimensionality) and delivers higher compression ratios, but requires custom kernel support that GQA does not.

• For long-context serving at $S \geq 8 , 000$, MLA is the current state-of-the-art architectural approach to keeping the KV cache manageable without sacrificing model quality.

16.2 Cross-Layer and Local Attention

16.2.1 Cross-Layer KV Sharing

You have already deployed GQA and your KV cache is still the binding memory constraint at large batch sizes or long context. The follow-up question in the interview, almost inevitably, is: what else can you do without changing the number of KV heads or the per-head dimension? Cross-Layer Attention (CLA) is the answer that shows up at senior and staff-level design rounds. It is conceptually simple, the math is one line, and it produces cache reductions that compound multiplicatively with GQA.

In a standard transformer, every layer $\ell$ has its own key and value projection matrices $W_{K}^{\ell}$ and $W_{V}^{\ell}$ . During generation, each layer writes its own KV slice to the cache, so total KV memory scales as $L \times T \times N_{k v} \times h \times 4$ bytes (layers $\times$ tokens $\times$ KV heads $\times$ head dim $\times$ K-and-V-in-bf16). The layer count $L$ multiplies every other factor: cutting it is a direct win.

CLA groups layers into blocks of size $G$ and forces all layers in a group to share the projection matrices of the first (leader) layer. For a group starting at layer $g_{0}$, every layer $\ell \in \{ g_{0} , g_{0} + 1 , . . . , g_{0} + G - 1 \}$ computes:

At inference time, the KV vectors are computed once for the group and reused. Layers $g_{0} + 1$ through $g_{0} + G - 1$ point back to the leader’s cache slice; no new memory is allocated for them. The query projections $W_{Q}^{\ell}$ remain independent per layer, so the model retains its full expressive capacity on the query side.

The cache reduction is exact: instead of $L$ independent KV slices you store $L / G$, cutting KV memory by a factor of $G$.

CLA does for the depth dimension what GQA does for the head dimension: it amortizes the KV cost through sharing, and the two techniques compose multiplicatively.

The intuition draws on the same logic as GQA. Adjacent layers in a deep residual network process information at similar levels of abstraction. The attention pattern - which positions a query attends to - is largely determined by key-vector content, and the structure relevant at layer $\ell$ is not radically different from the structure at layer $\ell - 1$. Empirically, models trained with sharing ratio $G = 2$ show negligible perplexity degradation; $G = 4$ begins to hurt on tasks that demand precise long-range recall.

The trade-off to name explicitly is expressiveness versus memory. Value vectors encode the information mixed into each output, so sharing them means higher layers in a group cannot independently learn what content to retrieve - only what positions to query. This is the binding constraint that limits how aggressively you can push $G$. In practice, $G = 2$ sits comfortably on the Pareto curve; beyond $G = 4$ expect accuracy regressions on retrieval-heavy benchmarks.

CLA contrasts with MLA (Multi-Head Latent Attention), which shrinks KV size by low-rank compression along the head-dimension axis. CLA instead reduces the number of independent KV slices across depth. They are complementary: you can stack CLA on top of MLA for compounded savings.

A Worked Example

Consider a 32-layer 7B model: $d = 4 , 096$, $N_{k v} = 32$ KV heads (standard MHA, no GQA), head dimension $h = 128$, bf16 weights and activations.

Per-token KV bytes across all 32 layers:

$32$ layers $\times 32$ heads $\times 128$ dim $\times 2 ($K and V$) \times 2$ bytes $= 524 , 288$ bytes $\approx 512$ KB per token.

At $T = 8 , 192$ tokens per sequence: $512$ KB $\times 8 , 192 = 4$ GB per sequence. At batch size $B = 32$: $4$ GB $\times 32 = 128$ GB, which overflows a single H100’s 80 GB HBM.

With CLA at $G = 2$ (16 independent KV groups):

$512$ KB$\times 8 , 192 = 2$ GB per sequence $\Rightarrow 64$ GB at $B = 32 . ( 2 \times$ reduction.$)$

Adding GQA-8 (reduce $N_{k v}$ from 32 to 8, a $4 \times$ reduction):

$512$ KB$\times 8 , 192 = 512$ MB per sequence $\Rightarrow 16$ GB at $B = 32 . ( 8 \times$ combined.$)$

The same configuration that saturated a single H100 now fits with 64 GB to spare, enabling larger batches and proportionally higher throughput.

What You Actually Decide in Practice

• Choose $G = 2$ by default. It halves KV memory with no statistically detectable accuracy loss on most benchmarks. Only push to $G = 4$ when memory constraints are extreme and the workload is retrievallight. Going above $G = 4$ is usually not worth it.

• Always make the first layer in the group the leader. The later layers in a group receive richer representations from the deeper residual stack; their KV projections would carry more nuanced content. Using the shallower layer’s projection is the conservative choice for accuracy.

• Scale the learning rate for shared projections. Because $W_{K}^{g_{0}}$ and $W_{V}^{g_{0}}$ receive gradient contributions from every layer in the group, the effective gradient magnitude is $G \times$ larger than for unshared projections. Either divide their learning rate by $G$ or put them in a separate optimizer parameter group with the adjusted rate.

• CLA and GQA compound multiplicatively. CLA-2 plus GQA-8 on the 32-layer 7B model above gives $8 \times$ reduction. Adding MLA-style KV compression on top of both can push beyond $10 \times$ versus vanilla MHA.

• RoPE compatibility is not an issue. RoPE is applied to queries and keys before the dot product, so the values stored in the KV cache already have position baked in. Reusing those vectors in neighboring layers is safe because the token sequence - and therefore the positional context - is identical across layers.

How This Shows Up in Interviews

• “You’ve deployed GQA on your serving cluster. What other architectural lever reduces KV cache footprint without touching head count or dimension?” Name CLA, explain the sharing across layers rather than across heads, and quantify: halving the number of independent KV slices halves cache memory, compounding with any GQA already in place. Mention the $G = 2$ rule of thumb.

• “What are the accuracy risks of CLA and how do you catch them in evaluation?” Cover the expressiveness trade-off on the value side. Cite $G = 2$ as safe and $G = 4$ as the threshold where you start checking retrieval-heavy benchmarks (needle-in-haystack, multi-hop QA). Those tasks are the canary; if they hold, general language modeling will too.

• Senior follow-up: “How does CLA interact with tensor parallelism and disaggregated KV caching?” With tensor parallelism, each device holds a shard of the KV heads. The shared KV shard still lives on its designated device; downstream layers in the group simply read it there instead of computing a fresh projection. Communication overhead per decode step decreases by $G$ because you issue one KV-projection communication per group rather than one per layer. In disaggregated serving, where the KV cache is offloaded to CPU or remote memory, CLA reduces the volume transferred back on every decode step by $G$ - since that transfer is often the dominant latency term in disaggregated architectures, the benefit is disproportionately large.

Key Takeaways

• CLA shares KV projection weights and cached vectors across a group of $G$ consecutive layers, cutting KV memory by exactly a factor of $G$ for those shared dimensions.

• $G = 2$ is the safe default; beyond $G = 4$, perplexity on recall tasks degrades noticeably and the gain stops being worth it.

• CLA is orthogonal to GQA and MLA - the three techniques stack multiplicatively on top of each other.

• Shared projections receive $G \times$ larger gradients; compensate with a $1 / G$ learning rate scaling for those parameter groups.

• CLA-2 combined with GQA-8 achieves an $8 \times$ KV reduction on a standard 32-layer model, turning an overflowing 128 GB cache into a manageable 16 GB on a single H100.

16.2.2 Sliding-Window and Hybrid Attention Layers

Even after deploying GQA and CLA, a model supporting 128K-token contexts faces a memory wall. A 32- layer model with 8 KV heads and head dimension 128, in bf16, stores $32 \times 8 \times 128 \times 4 = 131 , 072$ bytes of KV per token per sequence: at $T = 131 , 072$ that is roughly 17 GB per sequence, exhausting an H100 before you fit a single request. Sliding-window attention, combined with periodic full-attention layers in a hybrid architecture, is the mechanism frontier models use to serve contexts of 128K to 1M tokens without that blowup. This shows up in senior design rounds under any question framed as “design a long-context serving system” or “how does LLaMA 4 support 10M-token contexts.”

In standard causal attention, the query at position $t$ attends to all past tokens $\{ 0 , 1 , . . . , t - 1 \}$. The KV cache for that layer grows without bound as $T$ increases. Sliding-window (local) attention restricts each query to the most recent $W$ tokens:

The KV cache for a local layer never exceeds $W$ entries per layer, regardless of $T$. Once the sequence passes position $W$, old KV entries evict from the sliding window. Cache memory per local layer is a constant $W \times N_{k v} \times h \times 4$ bytes - no dependence on sequence length.

A pure sliding-window model has no long-range recall: it cannot attend to any token more than $W$ positions back. The fix is the hybrid architecture: place one full (global) attention layer every $k$ local layers. Global layers maintain the full $T$-length KV cache and handle long-range dependencies; local layers handle short-range coherence cheaply and constitute the majority of the stack.

Hybrid attention converts a hard memory cliff into a tunable dial: you pay full-attention cost only on $1 / k$ of your layers, and you choose $k$ based on how much long-range recall your workload demands.

The motivating fact is empirical: the overwhelming majority of next-token predictions depend on the last few hundred to few thousand tokens. The hard cases - retrieving a fact stated 50K tokens earlier, resolving a coreference across a long document - are a small fraction of all generation steps. A handful of global layers scattered through the stack handles them without forcing every layer to maintain the full context. On standard language modelling benchmarks, replacing five out of every six attention layers with local layers produces nearly identical perplexity, while peak KV memory drops by $3$-$5 \times$ at 32K context and $10 \times$ or more at 128K.

The alternative is learned sparse attention (Longformer-style dilated windows, block-sparse patterns). Those are more expressive for specialized tasks but require custom CUDA kernels and break compatibility with standard FlashAttention implementations. Hybrid attention is trivial to implement: the only change between a local and a global layer is the range of KV pairs fetched. You get the memory savings with no new infrastructure.

A second-order benefit appears at training time. Local layers have $O ( T \cdot W )$ attention cost rather than $O ( T^{2} )$, so training on long sequences becomes far more tractable, enabling the longer context windows that the serving system then exploits.

A Worked Example

Model: 32 layers, $N_{k v} = 32$ KV heads, head dim $h = 128$, bf16. Per-layer per-token KV bytes: $32 \times 128 \times 2 ($K+V$) \times 2$ bytes $= 16 , 384$ bytes.

Full attention at $T = 32 , 768$:

Hybrid (27 local layers at $W = 4 , 096$, 5 global layers at full $T$):

Notice that the local-layer cost stays fixed as $T$ grows - only the global layers scale. The reduction ratio improves with context length, exactly where it matters most. Combining this hybrid model with GQA-8 reduces the global-layer cost by a further $4 \times$, bringing the 128K total to approximately $3 . 4$ GB - a single H100 can now host multiple concurrent long-context sequences.

What You Actually Decide in Practice

• Global layer frequency $k$: Use $k = 4$ (one global per four layers) as a conservative default. Gemma and Mistral variants have used $k = 6$ successfully on standard tasks, but needle-in-haystack and multi-hop retrieval benchmarks tend to regress at $k = 6$ unless the global layers retain the full KV cache. Do not go coarser than $k = 6$ without ablation data on your target task distribution.

• Local window size $W$: $4 , 096$ is the common default and works well for most natural language and code tasks. Drop to $2 , 048$ only for workloads that are demonstrably short-range (single-sentence generation, classification). For multi-file code generation where function signatures and imports span many thousands of tokens, prefer $W = 8 , 192$.

• The no-RoPE trick on global layers (LLaMA 4-style): Apply RoPE only to local layers; use position-free full attention on global layers. Without a positional encoding, the global layer’s attention scores depend purely on content similarity, not distance. This makes the model near-insensitive to absolute token position on the global pathway, enabling context-length extrapolation well beyond the training window without ALiBi or positional interpolation. Use this when your serving target context is significantly longer than your training context.

• Combining with CLA: Apply cross-layer KV sharing within each run of local layers. Five consecutive local layers in a group can be CLA-2 paired, cutting their already-capped $W$-entry caches by a further factor of 2. The savings are additive in log-memory terms and require no changes to the global layers.

• Eviction strategy: The default is FIFO - evict the oldest token when the window is full. For tasks with multiple salient anchors spread across the context (legal document review, multi-turn dialogue with backreferences), “heavy hitter” eviction - retain the $W / 2$ most recently attended-to positions plus the $W / 2$ most recent positions - materially improves recall quality at the same cache budget.

How This Shows Up in Interviews

• “Design a system to serve a 70B model at 128K context on two H100s with reasonable throughput. How do you handle the KV cache?” Start from the memory budget ($2 \times 80$ GB minus $\approx 140$ GB model weights in bf16 leaves $\approx 20$ GB for KV). Reach for hybrid attention as the architectural lever. Name the window size and global-layer ratio you would pick, show the arithmetic (12 GB for a 32-layer model at 128K in the worked example above), then note that GQA-8 on the global layers brings that below 5 GB, making the budget work.

• “How does sliding-window attention affect tasks that require retrieving a fact stated 50K tokens ago?” Local layers cannot attend that far back; global layers can. The retrieval quality depends on whether there are enough global layers and whether those layers have the relevant token in their KV cache - which they do, because global layers retain full $T$ entries. Latency on those retrievals is no worse than standard attention, since global layers run a full attention pass regardless. The risk is that with only 1-in-6 global layers, some of those layers may not be the ones queried for that particular output position; whether that matters depends on where in the stack the “right” attention pattern emerges.

• Senior follow-up: “Explain the LLaMA 4 no-RoPE global layer and its failure modes.” Without positional encoding, the global layer treats all positions as equidistant and attends purely on content similarity. This enables extrapolation well beyond the training context because there is no monotonically decaying positional bias to interfere with distant attention. The failure mode is false-positive retrieval: the model may attend to a content-similar but irrelevant token far away in preference to the correct nearby token, because nothing in the attention score penalises distance. This shows up as degraded performance on needle-in-haystack benchmarks with high-similarity distractors. The mitigation is to keep global layers sparse enough that they are reserved for genuine long-range retrieval needs, rather than routine next-token coherence.

Key Takeaways

• Sliding-window attention caps each local layer’s KV cache at $W$ entries regardless of sequence length, converting $O ( T )$ memory into $O ( W )$ - a constant.

• Hybrid architectures place 1 global layer per $k$ local layers ($k = 4$ is conservative; $k = 6$ is aggressive). Global layers handle long-range dependencies and grow with $T$, but they are few enough to keep total memory practical.

• At $T = 128$K, a 32-layer hybrid model (5 global, 27 local at $W = 4 , 096$) uses roughly $5 . 5 \times$ less KV memory than full attention; the gap widens at longer contexts.

• The no-RoPE trick on global layers (LLaMA 4-style) removes positional bias from the long-range pathway, enabling near-unlimited context extrapolation at the cost of increased susceptibility to content-similar distractors.

• Hybrid attention and CLA compose cleanly: apply CLA within local-layer groups for multiplicative additional savings on top of the window-size savings from locality.