Chapter 19 Compression Techniques for Deployment

19.1 Quantization

19.1.1 INT8 Quantization with Outlier Handling (LLM.int8())

Imagine you have trained a 70B parameter model and now you need to serve it. At BF16, those weights consume 140 GB of memory-more than a single H100’s 80 GB HBM. INT8 would halve that to 70 GB, letting you serve on a single node. But every engineer who has tried naively packing a large transformer into INT8 has watched perplexity spike and downstream task accuracy collapse. Understanding why that happens, and how to fix it, is a question I have seen asked at nearly every frontier lab interview touching inference or deployment.

Quantization maps a floating-point tensor to a lower-precision integer representation. The simplest scheme, absmax quantization, works as follows: given a weight vector $\mathbf{w} \in \mathbb{R}^{n}$ stored in FP16, compute the scale $s = \max ( | \mathbf{w} | )$ and produce the INT8 vector

\mathbf{w}_{q} = \mathrm{round} \Bigl( \frac{\mathbf{w}}{s} \cdot 127 \Bigr) , \mathbf{w}_{q} \in [ - 127 , 127 ] .

To dequantize: $\hat{\mathbf{w}} = \mathbf{w}_{q} \cdot s / 127$ . Memory shrinks by $2 \times$ (2 bytes per weight in BF16, 1 byte in INT8). Bandwidth shrinks by the same factor, which directly translates to lower generation latency in the memory-bound regime.

The catch is outlier activations. Research has shown that in transformer models at or above 6.7B parameters, a small fraction of hidden-dimension channels-typically fewer than 1% of all feature dimensions-develop systematically large magnitudes, often 10-100 $\times$ larger than the median channel. These outliers emerge consistently across every token processed by the model. When you include even one such outlier in the absmax scale computation, the quantization grid becomes too coarse for every other channel, and the accumulated rounding error destroys model quality.

The LLM.int8() paper addresses this with a mixed-precision decomposition. Given the weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and the activation matrix $X \in \mathbb{R}^{B \times d_{\text{in}}}$ , partition the columns corresponding to outlier channels into an index set $\mathcal{O}$ (typically $| \mathcal{O} | < 0 . 01 \cdot d_{\text{in}}$ ). Then:

X W = X_{\mathcal{O}} W_{\mathcal{O}}^{\top} + X_{\bar{\mathcal{O}}} W_{\bar{\mathcal{O}}}^{\top} ,

where the first term is computed in FP16 (covering outlier columns) and the second in INT8 (covering the vast majority). The two partial products are accumulated back into FP16. Because outlier columns are identified offline from a small calibration set, no runtime search is required.

INT8 quantization at scale requires explicit outlier handling: a tiny fraction of channels disrupts the entire quantization grid, and the fix is to route those channels through a separate FP16 compute path.

The key property that makes the decomposition practical is that outlier channels are consistent across tokens. The same handful of columns that blow up in activation magnitude for one input tend to blow up for all inputs processed by a given layer. That means you can profile a few hundred calibration samples, identify the outlier channel indices once, and bake the decomposition into your serving kernel. There is no per-token routing overhead.

The trade-off is speed versus capacity. Because the FP16 path involves a sparse scatter-gather across outlier columns, LLM.int8() is typically 10-20% slower at runtime than a native BF16 forward pass on the same hardware. It is not a speed optimization-it is a memory-capacity optimization. The value proposition is: run a 175B model on 8 $\times$ 80 GB A100s instead of 16, or a 70B model on a single 8 $\times$ 80 GB node instead of two.

Compared to pure INT8 without outlier handling, mixed-precision decomposition recovers most of the quality loss (within 1-2 points on common benchmarks) at the cost of that modest throughput penalty. Compared to staying in BF16, you trade 10-20% throughput for a 2 $\times$ reduction in memory footprint and the ability to fit a larger model or serve a larger batch.

A Worked Example

Consider a single linear layer in a 13B model: $d_{\text{in}} = d_{\text{out}} = 5120$ , batch size $B = 16$ , BF16. The weight matrix $W$ is $5120 \times 5120$ , consuming $5120 \times 5120 \times 2 \approx 52$ MB per layer. Across 40 transformer layers (MLP up-projection, down-projection, and the three attention projections-roughly five large matrices per layer), total weight memory is around 26 GB, consistent with a 13B BF16 model.

After profiling 128 calibration samples, we identify 26 outlier columns (roughly 0.5% of 5120). These 26 columns remain in BF16: $26 \times 5120 \times 2 = 0 . 27$ MB per matrix. The remaining $5094$ columns are quantized to INT8: $5094 \times 5120 \times 1 = 26$ MB. Total per-matrix: $26 . 3$ MB versus the original 52 MB-a near $2 \times$ reduction with 99.5% of the compute running in INT8.

At inference, the forward pass for this layer does:

Load the 26-column FP16 slice of $W$ and the corresponding slice of $X$ ; compute $X_{\mathcal{O}} W_{\mathcal{O}}^{\top}$ in FP16.
Load the remaining 5094 columns as INT8; dequantize on-the-fly; compute $X_{\bar{\mathcal{O}}} W_{\bar{\mathcal{O}}}^{\top}$ in INT8.
Add the two partial products in FP16 to produce the output.

Total bytes transferred per token: $\approx 26$ MB (INT8 path dominates), versus 52 MB in BF16-halving memory bandwidth, which maps directly to halved generation latency per layer at batch size 1.

What You Actually Decide in Practice

Outlier threshold. A practical heuristic is to flag columns whose maximum absolute activation magnitude exceeds 6 standard deviations above the layer mean across your calibration set. Too low a threshold wastes compute on FP16; too high and you leave genuine outliers in the INT8 path.
LLM.int8() is a capacity tool, not a speed tool. If your model fits comfortably in BF16, do not use it. Use it when you are trying to serve a model on fewer GPUs or to increase maximum batch size.
Calibration data quality matters. Outlier detection is distribution-sensitive. A calibration set drawn only from short chat exchanges will misidentify outliers when the model processes long code contexts. Use diverse calibration data representative of your actual traffic.
Kernel support. The mixed-precision path requires custom CUDA kernels (bitsandbytes is the canonical open-source implementation). Naively implementing this in standard PyTorch will not give you the memory savings because PyTorch may still keep the original FP16 weights around.
Interaction with GQA and MLA. Outlier patterns differ across projection types. KV projections in models using grouped-query attention typically have fewer outliers than the value projections in the MLP. Profiling each projection separately rather than using a single global threshold improves quality.

How This Shows Up in Interviews

“A 70B model is too large for your H100 fleet. How do you fit it?” A strong answer names three levers: model parallelism (tensor or pipeline), lower precision (INT8, INT4), and quantization schemes (LLM.int8() for memory-safe INT8). The candidate should quantify: 70B at BF16 is 140 GB; at INT8 it is 70 GB, fitting on one 8-way H100 node.
“What goes wrong when you naively quantize a large transformer to INT8?” Strong answer: outlier activations in 1% of channels inflate the absmax scale, making the INT8 grid too coarse for the remaining 99% of channels. The fix is mixed-precision decomposition for outlier columns.
“Why do outliers appear more in larger models?” This is the senior follow-up. The empirical finding is that outlier channels emerge at model scales of roughly 6.7B parameters and above, growing more extreme with scale. The intuition is that larger models rely on specific feature dimensions to carry sharp, high-magnitude signals-a form of emergent specialization. A senior answer connects this to the phase transitions observed in scaling laws and notes that the outlier columns correspond to attention heads or MLP neurons that have learned to function as routing or gating mechanisms.

Key Takeaways

INT8 absmax quantization halves weight memory but causes severe accuracy degradation in models at or above 6.7B parameters due to outlier activation channels.
LLM.int8() uses mixed-precision decomposition: $\sim$ 1% of columns in FP16, the rest in INT8, identified offline from a calibration set.
The method is a capacity optimization-it enables fitting larger models in GPU memory-not a throughput optimization; it typically runs 10-20% slower than native BF16.
Outlier channels are stable across inputs, making offline profiling viable for production deployments.
Use LLM.int8() when memory is the binding constraint; move to AWQ or INT4 schemes when you also need throughput gains.

19.1.2 Activation-Aware Weight Quantization (AWQ) and INT4

Once LLM.int8() gets your 70B model into memory, the next interviewer question is: “Now make it fast.” LLM.int8() is a capacity tool-it typically runs 10-20% slower than BF16 because the mixed-precision scattergather adds overhead. AWQ (Activation-Aware Weight Quantization) targets a different objective: squeezing throughput by going to INT4, which halves memory bandwidth again relative to INT8 and delivers 2-3 $\times$ end-to-end generation speedup over BF16 with minimal accuracy loss. This question shows up when an interviewer has already confirmed you can fit the model and wants to know whether you understand the path to production throughput targets.

The core insight of AWQ is that weights are not equally important for quantization error. Given a linear layer $Y = X W$ , a small fraction of input channels-those multiplied by large activation magnitudes-contribute disproportionately to the output. When you quantize a salient weight channel $j$ coarsely, the output error is $\Delta Y_{j} = x_{j} \cdot \Delta w_{j}$ . If $x_{j}$ is large, even a small per-weight quantization noise $\Delta w_{j}$ causes significant output error. Non-salient channels with small $x_{j}$ can absorb much larger $\Delta w_{j}$ without visible effect.

AWQ identifies salient channels by running a small calibration set through the model. For each input channel $j$ , compute the average activation magnitude:

\bar{s}_{j} = \mathbb{E}_{x \sim \mathcal{D}} [ | x_{j} | ] .

Channels with large $\bar{s}_{j}$ are salient. Rather than routing them to a separate FP16 compute path (the LLM.int8() strategy), AWQ applies a per-channel scale $s_{j} \propto \bar{s}_{j}^{\alpha}$ to the weights before quantization and compensates by dividing activations by the same scale at runtime:

XW=\underbrace{\bigl(X\cdot\operatorname{diag}(\mathbf{s})^{-1}\bigr)}_{X^{\prime}}\cdot\underbrace{\bigl(\operatorname{diag}(\mathbf{s})\cdot W\bigr)}_{W^{\prime}},

where $W^{\prime}$ is the scaled weight matrix that gets quantized to INT4, and the factor $\mathrm{diag} ( \mathbf{s} )^{- 1}$ on the activation side is folded into the preceding layer normalization or fused into the preceding kernel-it costs nothing extra at runtime. By setting $\alpha \approx 0 . 5$ , salient channels are scaled up, giving them finer quantization resolution, while non-salient channels are barely affected.

In practice, weights are quantized in groups: every 128 consecutive weights share one FP16 scale and zero-point, a scheme called group quantization with $g = 128$ . This prevents a single outlier weight from degrading an entire row and is the dominant production choice. The effective bit-width is slightly above 4 (128 INT4 values plus one FP16 scale and zero-point per group $\approx$ 4.25 bits per weight) but the storage overhead is negligible.

AWQ finds which weights matter most, scales them up before INT4 quantization so the grid is finer where errors count, and absorbs the correction into adjacent layers for free.

The bandwidth argument is direct. Inference generation is bandwidth-bound: every generation step loads the full weight matrix once to multiply against a single token. Cutting weight precision from BF16 (2 bytes) to INT4 (0.5 bytes) reduces bytes transferred per layer by $4 \times$ . In the pure bandwidth limit that translates to $4 \times$ lower latency per token. Real systems achieve 2-3 $\times$ because of kernel launch overhead, the cost of dequantizing INT4 to FP16 on the fly, and the residual bandwidth of FP16 group scales.

AWQ’s accuracy advantage over naive INT4 comes from the weighted error reduction. Scaling $w_{j}$ by $s_{j}$ before quantization makes the post-dequantization noise $\Delta w_{j}^{\prime} = \Delta w_{j} / s_{j}$ , so the output error for salient channel $j$ drops from $x_{j} \Delta w_{j}$ to $x_{j} \Delta w_{j} / s_{j}$ -reduced by exactly the scale factor. Non-salient channels with small $x_{j}$ are unaffected because their output error was already small. AWQ searches the exponent $\alpha$ over a coarse grid (0.0 to 1.0 in steps of 0.1) to minimize calibration-set quantization error, making the procedure fast and calibration-data-efficient.

The design contrast with LLM.int8() is worth internalizing. LLM.int8() targets INT8 weights and handles activation-outlier channels with a separate FP16 compute path; the result is a model that fits in memory but does not run faster. AWQ targets INT4 weights only, leaving activations in FP16 (the W4A16 regime). W4A16 is simpler-weight quantization is far more tractable than activation quantization because weights are static-and it targets throughput rather than just capacity.

A Worked Example

Consider a 7B model: $d = 4096$ , 32 transformer layers, MLP hidden dimension $F = 11008$ , BF16 baseline.

Memory. BF16 weight footprint: $7 \times 10^{9} \times 2 \approx 14$ GB. After AWQ INT4 with group size 128: $7 \times 10^{9} \times 0 . 5 = 3 . 5$ GB for INT4 weights, plus $7 \times 10^{9} / 128 \times 2 \approx 0 . 11$ GB for FP16 group scales. Total: $\approx 3 . 6$ GB-a $3 . 9 \times$ reduction.

Per-layer bandwidth. Take the MLP up-projection: $W \in \mathbb{R}^{11008 \times 4096}$ . BF16 storage: $11008 \times 4096 \times 2 = 90 . 2$ MB. AWQ INT4 storage: $11008 \times 4096 \times 0 . 5 = 22 . 5$ MB for INT4 weights, plus $11008 \times 4096 / 128 \times 2 =$ $0 . 7$ MB for scales. Total: $\approx 23 . 2$ MB.

Latency per token, H100 (3.35 TB/s HBM bandwidth):

t_{\text{BF16}} = \frac{90 . 2 \text{MB}}{3 . 35 \text{TB/s}} \approx 27 \mu \text{s} , \qquad t_{\text{AWQ}} = \frac{23 . 2 \text{MB}}{3 . 35 \text{TB/s}} \approx 7 \mu \text{s} .

Across 32 layers and five large weight matrices per layer (Q, K, V, O projections plus the MLP stack), the pertoken generation step drops from $\approx$ 4.3 ms to $\approx$ 1.1 ms in the bandwidth limit-roughly $3 . 9 \times$ . Accounting for dequantization kernel overhead and the FP16 activation path, measured throughput on published benchmarks sits at 2.5-3 $\times$ , consistent with the theoretical upper bound.

What You Actually Decide in Practice

Group size is your accuracy-vs-speed knob. Group size 128 is the standard: it limits scale overhead to $< 2 \%$ of total weight storage and keeps accuracy within 1-2 points of BF16 on most benchmarks. For models under 3B parameters, tighten to group size 64 to compensate for the stronger relative compression. Widening to 256 saves a bit more memory but hurts accuracy noticeably on code and math tasks where precision matters.
W4A16 before W4A8. Quantizing activations (W4A8) further reduces compute cost but requires pertoken activation scales that must be computed at runtime, adds latency at low batch sizes, and is sensitive to input distribution shifts. Start with W4A16, validate accuracy on your eval suite, and consider W4A8 only if you have a hardware-specific reason (e.g., INT8 tensor cores running at $2 \times$ the FLOP rate of FP16 on your device).
Do not quantize embedding tables or the LM head. These layers are accessed once per forward pass in a different access pattern and are small relative to the total parameter count. Quantizing them saves negligible bandwidth and carries real accuracy risk for low-frequency tokens whose embeddings can be hard to represent in INT4.
Calibration data should match your serving distribution. AWQ activation magnitude estimates stabilize at 128-512 samples. Use diverse data spanning your expected input types-chat, code, documents-so salient channels are correctly identified for all input modes. A calibration set drawn purely from short chat turns will misidentify salience for long-context or structured-output use cases.
Stack AWQ on top of GQA for maximum throughput gain. GQA already reduced KV-projection head counts; AWQ cuts the projection weight bandwidth by another $4 \times$ . The two compose multiplicatively on the KV projection layers and are independent on MLP layers. A LLaMA-3-70B model with GQA and AWQ INT4 can serve on a single 8 $\times$ 80 GB H100 node with throughput competitive with a BF16 LLaMA- 3-7B on the same hardware.

How This Shows Up in Interviews

“We need $3 \times$ throughput improvement on a production 7B model without retraining. What is your plan?” Strong answer: quantize weights to INT4 using AWQ with group size 128. Generation is bandwidth-bound; INT4 cuts weight bandwidth by $4 \times$ relative to BF16; AWQ’s activation-magnitudebased scaling preserves accuracy within 1-2 points. Expect 2-3 $\times$ real-world speedup with near-zero accuracy loss on standard benchmarks.
“What is the difference between LLM.int8() and AWQ?” Strong answer: LLM.int8() is a capacity tool-it targets INT8 with a mixed FP16/INT8 split on activation-outlier channels; it fits models that do not otherwise fit in memory but typically runs slightly slower than BF16. AWQ is a throughput tool-it targets INT4 weight-only quantization guided by activation magnitudes; it genuinely makes generation faster by reducing bandwidth. They solve different problems and can coexist (though INT4 alone usually dominates once both goals are met).
Senior follow-up: “Why does scaling weights by activation magnitude before quantization actually reduce output error? Walk me through the math.” Strong senior answer: the per-token quantization output error for channel $j$ is $\epsilon_{j} = x_{j} \cdot \Delta w_{j}$ , where $\Delta w_{j} \approx \| w_{j} \|_{\infty} / 127$ (the INT8 step size). By premultiplying $w_{j}$ by $s_{j} > 1$ before quantization, the post-dequantization weight error becomes $\Delta w_{j}^{\prime} =$

$\Delta w_{j} / s_{j}$ , reducing output error to $\epsilon_{j}^{\prime} = x_{j} \cdot \Delta w_{j} / s_{j}$ . Setting $s_{j} \propto \bar{s}_{j}^{\alpha}$ with $\alpha \approx 0 . 5$ balances error reduction for salient channels against the distortion imposed on non-salient channels. The $\alpha$ search is cheap: it is a 1D grid search over ten values using only the calibration set, taking seconds per layer.

Key Takeaways

AWQ identifies weight channels that matter most by their corresponding activation magnitudes, scales them up before INT4 quantization for finer resolution, and absorbs the inverse scale into the preceding layer at zero runtime cost.
INT4 weight quantization with group size 128 (W4A16) cuts weight memory by $\approx 4 \times$ versus BF16 and delivers 2-3 $\times$ practical generation throughput improvement by reducing bandwidth in the memory-bound generation regime.
Group quantization is non-negotiable at INT4: without it, a single outlier weight degrades the entire row’s resolution, causing accuracy collapse on long-tail tokens.
AWQ is a throughput optimization; LLM.int8() is a capacity optimization. Use AWQ when you need faster tokens per second from a model that already fits in memory; use LLM.int8() when memory is the binding constraint.
AWQ and GQA compose multiplicatively on KV projections; combining both is a practical path to serving 70B-class models at 7B-class latency on a single node.

19.2 Pruning and Distillation

19.2.1 Structured Pruning of Layers, Heads, and Channels

In production, the pressure to cut serving costs often arrives after a model is already deployed and has real traffic. The question is not “can we train a smaller model?” but “can we make this specific 15B checkpoint cheaper without starting over?” Structured pruning is the tool that answers yes: remove entire transformer layers, attention heads, or MLP hidden channels from a trained model and hand back a smaller, denser model that runs on unmodified kernels with genuine latency and memory savings. This topic comes up regularly in senior inference interviews because it sits at the intersection of model architecture, importance estimation, and deployment economics.

Unstructured pruning zeros individual weights scattered throughout a weight matrix. Modern hardware cannot exploit that sparsity efficiently-you still load every weight from HBM before zeroing it in registers, so you get no memory bandwidth reduction and no throughput gain. Structured pruning removes entire contiguous blocks: a full transformer layer, all positions of one attention head, or a set of MLP hidden-dim channels. The result is a model with genuinely smaller dense matrices, producing linear reductions in parameter count, memory footprint, and compute.

The standard three-step pipeline is:

Score. Run a small calibration set (typically 128-512 representative sequences) through the model and compute an importance score for each structural unit.
Remove. Zero out or drop the lowest-scoring units until you reach the target parameter budget.
Repair. Recover lost accuracy by fine-tuning or distillation from the original model (covered in the next subsection).

Several importance criteria are used in practice. The simplest is activation magnitude: compute the mean $\ell_{2}$ norm of a unit’s output across the calibration set-low-magnitude units contribute little to downstream residuals and are safe to remove. A more principled criterion is the first-order Taylor approximation to the loss increase caused by removing unit $u$ :

\Delta \mathcal{L}_{u} \approx \sum_{i \in u} | g_{i} \cdot w_{i} | ,

where $g_{i}$ is the gradient and $w_{i}$ the weight at position $i$ . For layer removal specifically, a cheap proxy is the cosine similarity between a layer’s input and output activation vectors: a layer whose output is nearly identical to its input adds almost nothing to the residual stream and is the first candidate for deletion.

Structured pruning converts a trained model into a smaller one that runs on unmodified kernels, trading a controlled accuracy loss for proportional reductions in parameter count, memory, and inference latency.

The alternative-training a smaller model from scratch-requires the full pre-training compute budget. A 15B model that took weeks on hundreds of GPUs cannot be reproduced in the time a product team has. Structured pruning plus distillation recovery delivers a comparable result in one to two orders of magnitude less compute.

Structured pruning also composes naturally with quantization. Quantization reduces bytes per weight (e.g., BF16 to INT4) but leaves the number of matrix operations unchanged. Pruning reduces the number of operations outright. Applying both-prune first to shrink layer dimensions, then quantize the surviving weights-stacks the gains without interference, as long as you run quantization calibration after pruning and not before.

The drawback compared to unstructured pruning is that structured pruning is coarser: you must remove entire heads or layers even if only a fraction of their weights are dispensable. This means a given parameter reduction target costs more accuracy points than unstructured pruning on the same model. The tradeoff is that hardware speedup from structured pruning is real and immediate, whereas the theoretical speedup from unstructured pruning is largely unrealized on current accelerators.

A Worked Example

Consider an NVIDIA-style compression of a 15B Llama-family model: 32 layers, $d_{\text{model}} = 5120$ , 40 attention heads, MLP hidden dim $d_{\text{ff}} = 13 , 696$ .

Step 1 - score layers. Run 256 sequences from a mixed pre-training corpus. For each of the 32 layers compute the mean cosine similarity between the residual stream at the layer’s input and output. Layers 14-18 (mid-network) consistently show cosine similarity above 0.97-nearly identity transformations. These are flagged for removal.

Step 2 - score heads and channels. For each attention head compute the mean output norm; for each MLP hidden channel compute the mean activation norm. Rank both globally.

Step 3 - remove to target. Remove 8 of the 32 layers (the 8 with highest input-output cosine similarity). Trim the MLP hidden dim from 13,696 to 9,216 by dropping the lowest-norm channels. Optionally reduce KV heads from 40 to 8 (GQA-style) for additional KV cache savings.

After pruning: approximately 8.1B parameters. MMLU immediately after pruning drops from 65% to roughly 58%-a 7-point raw gap. Memory for weights in BF16 shrinks from 30 GB to 16 GB, and generation throughput at batch size 32 on an H100 increases from $\approx 3 , 200$ tokens/s to $\approx 5 , 800$ tokens/s, a $1 . 8 \times$ improvement, before any distillation recovery.

After a short distillation pass (covered in the next subsection), MMLU recovers to roughly 64.5%-within half a point of the original 15B, at 53% of the parameter count and $1 . 8 \times$ the throughput.

What You Actually Decide in Practice

Prune layers before heads or channels. Layer removal is the highest-leverage decision: each removed layer eliminates $\approx 2 \times d_{\text{model}} \times d_{\text{ff}}$ parameters in one cut. Score layers first, remove them, then re-score heads and channels on the surviving architecture. Pruning all dimensions simultaneously compounds errors.
Score on calibration data that matches your deployment distribution. A calibration set that is too short ( $< 128$ sequences) or domain-mismatched produces noisy importance scores. If you prune a coding model with web-text calibration data, you will over-prune heads that are important for code structure and miss heads that are dead weight for your actual traffic.
Use cosine similarity for layer removal, Taylor sensitivity for head and channel removal. Cosine similarity is fast and requires no gradients. Taylor sensitivity is more accurate for fine-grained removal but requires a backward pass. This split gives you the best of both at reasonable cost.
Keep a validation set of difficult prompts and check it after each pruning step. Accuracy on standard benchmarks like MMLU is not sensitive enough to catch regressions in long-context or multi-step reasoning tasks. Run at least one specialized eval (e.g., a chain-of-thought benchmark) before committing to the pruned structure.
Never quantize before pruning. Some quantization calibration methods (AWQ-style) compute per-channel weight scales from activation statistics. If you prune channels after calibration, those scales are stale. The correct order is always: prune $\rightarrow$ distill $\rightarrow$ quantize.

How This Shows Up in Interviews

“How would you compress a 70B model to fit on a single 80 GB H100?” The 70B model in BF16 is $\approx 140$ GB, which does not fit. Structured pruning to $\sim$ 35B active parameters brings the weight footprint to $\sim$ 70 GB; add INT4 quantization to land at $\sim$ 35 GB with room for KV cache. Distill from the 70B teacher to recover accuracy. A strong answer names this sequence and gives the memory arithmetic.
“What is the difference between structured and unstructured pruning, and why does it matter for inference?” Unstructured pruning produces sparse weight matrices that do not reduce memory bandwidth on dense-matrix hardware. Structured pruning removes whole rows, columns, heads, or layers, producing smaller dense tensors with proportional speedup on standard kernels. This is the answer that separates a candidate who has thought about hardware from one who has only read papers.
Senior follow-up: “You removed 8 of 32 layers from a 15B model and see a 10-point MMLU drop. Before running distillation, what do you investigate?” First, break down the MMLU drop by category: if it is uniform, the issue is capacity loss that distillation can recover. If it is concentrated in world knowledge or multi-hop reasoning questions, you may have removed layers that serve as key associative recall steps. In that case, swap in the two or three layers with the highest Taylor sensitivity scores instead of the ones with the highest cosine similarity. Cosine similarity is a coarse proxy; for reasoning-heavy tasks, sensitivity-based scoring is more reliable.

19.2.2 Distillation Recovery from a Pruned Initialization

Structured pruning always leaves you with a model that is noticeably worse than the original. The question every interview panel eventually asks is: how do you get the accuracy back, and what does it cost? Knowledge distillation from the pruned initialization is the standard answer, and it shows up at every seniority level in LLM inference interviews because it bridges the architectural decision to prune and the quality guarantee you owe a product team. What separates a strong answer from a weak one is understanding why distillation from a pruned init is fundamentally different from-and cheaper than-training a small model from scratch.

Standard cross-entropy fine-tuning trains a model against hard one-hot labels: at each token position, only the ground-truth token contributes gradient. Knowledge distillation instead trains the student to match the teacher’s full probability distribution over the vocabulary. The training loss is the forward KL divergence between the teacher’s next-token distribution $p_{T}$ and the student’s $p_{S}$ :

\mathcal{L}_{\text{KD}} = \sum_{t} D_{\text{KL}} ( p_{T} ( x_{t} | x_{< t} ) \| p_{S} ( x_{t} | x_{< t} ) ) .

In practice, a small cross-entropy term is mixed in to prevent the student from ignoring the ground-truth signal entirely:

\mathcal{L} = \mathcal{L}_{\text{KD}} + \lambda \mathcal{L}_{\text{CE}} , \lambda \approx 0 . 1 .

The key insight is not the loss function-it is the initialization. When you prune a 15B model to produce an 8B student, the surviving weights are already the best 8B-worth of weights the teacher had. The student starts with a loss that is already near the pruning-induced gap, not at random initialization. Training from that point requires only a small correction, not a full optimization trajectory from scratch.

An optional extension is feature-level distillation: for each student layer $l$ paired with a corresponding teacher layer $l^{\prime}$ , add a loss term

\mathcal{L}_{\text{feat}}=\sum_l\left\|h_S^{(l)}-W_{\text{proj}}h_T^{(l^{\prime})}\right\|_2^2,

where $W_{\text{proj}}$ is a learnable linear projection to align dimensions when the teacher and student hidden sizes differ. This aligns intermediate representations and accelerates accuracy recovery but requires the teacher to remain resident in GPU memory for every training step.

Distilling from a pruned initialization is strictly better than training a small model from scratch because you are correcting a small perturbation in an already-converged loss landscape, not optimizing from a random starting point.

Training a fresh 8B model from scratch requires hundreds of billions to a trillion tokens-roughly the same scale as pre-training the original. Distillation from a pruned 15B initialization recovers most of the accuracy loss with far fewer tokens, because the student has already seen the effective gradient signal accumulated over the teacher’s full pre-training. NVIDIA’s Minitron experiments show recovery to within one MMLU point using roughly 128B tokens of distillation, compared to $\sim$ 1T tokens for scratch training to equivalent quality-a $\sim$ 8 $\times$ reduction in the recovery budget alone.

The soft probability targets from the teacher provide a dense per-step gradient signal. A one-hot label on a 32,000-token vocabulary gives exactly one nonzero gradient component per position. The teacher’s distribution typically assigns non-negligible probability to dozens of plausible continuations, each contributing gradient. This richness is especially helpful for recovering the student’s early layers, which must now handle token interactions that the pruned architecture routes differently than the original.

Versus other recovery strategies-continued pre-training, LoRA fine-tuning, task-specific fine-tuning-distillation is the only approach that recovers general-purpose capability rather than task-specific accuracy. Fine-tuning on a downstream task recovers performance on that task but does not restore the broad capability distribution of the original model. If you need a general-purpose compressed model, distillation is the right tool.

A Worked Example

Setup: teacher is a 15B Llama-family model; student is the 8.1B model produced by structured pruning as described in the previous subsection. Pre-pruning MMLU: 65%. Post-pruning MMLU: 58%. Target: recover to within 1 point of 65%.

Assemble the calibration corpus. Sample 128B tokens from the same distribution as pre-training-mixed web text, code, multilingual data-in the original mixture ratios. Do not use only downstream task data; you want to recover general capability, not overfit to a benchmark.
Configure the loss. Use output-level KD with temperature $\tau = 2$ (softer distributions help the student learn inter-token relationships). Set $\lambda = 0 . 1$ for the cross-entropy term.
Estimate the compute cost. Each distillation step requires:

One teacher forward pass: $\approx 2 \times 15$ B $\times T$ FLOPs per batch
One student forward pass and backward pass: $\approx 6 \times 8$ B $\times T$ FLOPs per batch Over 128B tokens with sequence length 4,096, total compute is roughly $6 \times 8 \times 10^{9} \times 128 \times 10^{9} \approx 6 \times 10^{21}$

FLOPs for the student, plus $\approx 2 \times 10^{21}$ for teacher inference. Compare to scratch training an 8B model on 1T tokens: $6 \times 8 \times 10^{9} \times 10^{12} \approx 5 \times 10^{22}$ FLOPs-roughly $6 \times$ more compute for the scratch run.

Results. After the 128B-token distillation pass:

MMLU: 64.5% (recovered 6.5 of the 7-point pruning gap)
Coding benchmark (MBPP): recovers from 47% to 54% against teacher 55%
Generation throughput at batch size 32: $\approx 5 , 800$ tokens/s vs. $\approx 3 , 200$ tokens/s for the original 15B-an $\approx 1 . 8 \times$ improvement The remaining 0.5-point MMLU gap is the irreducible cost of reduced capacity. Extending distillation to 256B tokens closes it by another 0.2 points but with sharply diminishing returns.

What You Actually Decide in Practice

Start with output-level KD; add feature-level distillation only for persistent gaps. Feature-level distillation keeps the teacher resident in memory for every training step, roughly doubling GPU memory pressure. In most cases, output-level KD over 50-100B tokens recovers 85-90% of the pruning loss. Only add the feature-level term if you see a systematic gap on specific capability categories that does not close with more output-level tokens.
Use temperature $\tau = 2$ - $4$ for the KD loss. Temperature $\tau = 1$ (raw logits) produces a distribution that is too peaked on the argmax, reducing the soft-label information. A temperature of 2-4 flattens the distribution enough that secondary tokens contribute meaningful gradient. A practical default is $\tau = 2$ with a linear warmup of the KD loss weight over the first 5% of training steps.
Match calibration data to your deployment traffic. The distillation step is short enough that domaintargeted data is tractable. If your compressed model will primarily serve code generation, distill on a code-heavy corpus. A general-purpose distillation that underperforms on your actual request distribution costs user quality on every inference for the model’s lifetime.
Expect 90% accuracy recovery in 10-100B tokens for models above 7B. Returns diminish sharply after 100B tokens because the student has effectively exhausted its capacity; additional tokens are not adding new gradient directions. If you are still seeing a large gap at 100B tokens, the gap is structural (too aggressive a pruning ratio), not a training-budget problem.
The prune-then-distill recipe breaks down above $\sim$ 50% parameter reduction. Removing more than half the parameters leaves the pruned initialization with too little useful structure. A fresh 8B model trained on the same data as a pruned-and-distilled 7.5B-from-15B will typically be competitive or better because it is not constrained to a damaged architecture. Treat 50% as the rough ceiling; beyond it, consider training a new small model from scratch using the large model’s outputs as training labels (offline distillation) rather than starting from a pruned checkpoint.

How This Shows Up in Interviews

“Walk me through producing an 8B model from a 15B checkpoint without full retraining.” Strong answer: run structured pruning to remove layers and channels using importance scores on a calibration set, then run output-level knowledge distillation from the 15B teacher over 50-100B tokens with $\tau = 2$ . Name the compute saving ( $\sim$ 6-8 $\times$ vs. scratch training) and the expected accuracy gap (within 1 MMLU point for a 15B $\rightarrow$ 8B compression).
“Why is distilling from a pruned initialization better than training a small model from scratch?” Strong answer: the pruned model starts with weights already optimized over the teacher’s full pre-training; it begins near the target loss rather than at random initialization. The distillation step corrects a small perturbation, not the full optimization problem. This translates to 6-8 $\times$ fewer training tokens to reach equivalent quality.
“What loss function do you use and why not just cross-entropy?” Strong answer: forward KL divergence between teacher and student distributions. Cross-entropy against hard one-hot labels discards the teacher’s uncertainty information; the soft distribution provides gradient from all high-probability tokens, not just the argmax. The richness of the signal is especially important in the early distillation steps when the student’s distribution is still far from the teacher’s.
Senior follow-up: “After distillation your 8B model recovers MMLU well but regresses significantly on multi-step reasoning. What is the most likely cause and how do you fix it?” Multi-step reasoning relies on the residual stream accumulating state across specific layers. If you removed layers uniformly by cosine similarity, you may have deleted layers that specialize in intermediate reasoning steps even if their input-output similarity happened to be high on general calibration data. Diagnosis: re-run the importance scoring using a calibration set that includes chain-of-thought traces and compare which layers are now flagged. Fix: restore the two or three reasoning-critical layers and re-prune a different set of lesscritical layers to maintain the parameter budget, then re-distill with a calibration corpus weighted toward reasoning-heavy tasks.