Chapter 4 Hyperparameters You Will Be Asked to Justify

4.1 Width and Depth

4.1.1 Aspect Ratio and the “128 Hidden Dims per Layer” Rule

When an interviewer says “design a 7B-parameter dense decoder model,” the next question is almost always: how many layers, and how wide? This ratio-what practitioners call the aspect ratio-is the single structural decision that cascades into your parallelism strategy, your per-device memory footprint, and your inter-node networking requirements. Mid-level engineers treat it as a free variable; senior engineers know the empirically validated sweet spot and can argue for it on both quality and systems grounds.

The aspect ratio is the ratio of the residual stream width to the number of transformer layers:

$\text{layers}$ It captures how a fixed parameter budget is allocated between width and depth. A model with $d_{\text{model}} = 4096$ and $n_{\text{layers}} = 32$ has aspect ratio $128$; redistributing the same parameter count to $d_{\text{model}} = 8192$ and $n_{\text{layers}} = 16$ gives aspect ratio $512$.

Hyperparameter sweeps across three model scales (50M, 274M, and 1.5B parameters) show that loss is minimized at aspect ratios between 100 and 128, and that this optimum is remarkably stable across scales. GPT-3 (175B, ratio 128), LLaMA-2 7B (ratio 128), LLaMA-2 13B (ratio 128), and LLaMA-2 70B (ratio 102) all land squarely in this band. That stability is the key finding: you can validate your aspect ratio at small scale and carry it forward to full training with reasonable confidence.

Width and depth are not equivalent ways to spend parameters. Width increases the rank and representational capacity of every vector in the residual stream. Depth increases the number of sequential nonlinear transformations applied. The aspect ratio controls their balance.

The rule of thumb: target roughly 100-128 hidden dimensions per layer, build your $d_{\text{model}}$ and $n_{\text{layers}}$ around that constraint, and you will be within the flat loss minimum for every scale that has been publicly studied.

Too narrow and too deep: each layer has low representational capacity, the residual stream bottlenecks information flow, and pipeline stages become tiny-wasting GPU utilization at stage boundaries. Too wide and too shallow: you over-invest in representation capacity at each step before enough nonlinear transformations can process it, and the very wide matrix multiplies require aggressive tensor parallelism that strains inter-GPU bandwidth.

The aspect ratio also directly determines which parallelism strategies are feasible. Wide models (high $d_{\text{model}}$) suit tensor parallelism, where each weight matrix is partitioned column-wise or row-wise across GPUs within a node. Tensor parallelism requires high-bandwidth, low-latency interconnects such as NVLink at $\geq 600$ GB/s. Deep models (many layers) suit pipeline parallelism, where consecutive layer groups are assigned to different GPU nodes; pipeline parallelism tolerates slower inter-node networking (200 Gbps InfiniBand is usually sufficient) because communication only occurs at stage boundaries. Your cluster topology therefore constrains the aspect ratio before model quality does.

One important nuance separates pretraining loss from downstream task performance. Research comparing depth versus width at matched parameter counts finds that pretraining perplexity is insensitive to the splittotal parameters dominate. But downstream fine-tuning accuracy can mildly favor deeper models at the same parameter count. If you optimize purely around pretraining loss, aspect ratio near 128 is well validated. If downstream task capability is the primary target, there is a weak case for going somewhat deeper, within the constraint that pipeline parallelism remains efficient.

A Worked Example

Consider designing a 7B-parameter dense decoder model. Three candidate configurations:

The standard configuration matches LLaMA-2 7B exactly. On a single 8-GPU H100 node with NVLink, it fits comfortably with 4-way tensor parallelism ($d_{\text{model}} /$TP $= 1024$ per shard, cleanly divisible). With 32 layers, pipeline parallelism across two groups of 4 GPUs assigns 16 layers per stage-a sensible granularity.

The wide variant ($d_{\text{model}} = 8192$, 14 layers) has very few pipeline stages, making coarse-grained pipeline parallelism inefficient. Effective tensor parallelism would need degree 8 or 16, requiring a dense NVLink fabric across multiple nodes-a significant hardware constraint. Attention with 64 heads at head dimension $8192 / 64 = 128$ is fine; the real cost is that wide matmuls do not automatically saturate hardware better unless the batch dimension is simultaneously large.

The narrow variant ($d_{\text{model}} = 2048$, 64 layers) creates very fine-grained pipeline stages. At 8 layers per stage across 8 pipeline stages, activation tensors moving between stages are small, and pipeline bubbles dominate runtime. Tensor parallelism across small matrices is unattractive because communication overhead relative to compute grows as matrix size shrinks.

The arithmetic conclusion: for a 7B model on 8 H100s, 4096 hidden dimensions and 32 layers produces a clean 4-way TP within each group of 4 GPUs and 2-way pipeline parallel across groups-no wasted hardware, minimal bubble fraction.

What You Actually Decide in Practice

• Start at aspect ratio 128. For a 7B model: $d_{\text{model}} = 4096$, $n_{\text{layers}} = 32$. For 13B: $d_{\text{model}} = 5120$, $n_{\text{layers}} = 40$. For 70B: $d_{\text{model}} = 8192$, $n_{\text{layers}} = 80$ (ratio $\approx 102$, close enough). These numbers match the released LLaMA-2 configs because the same reasoning applies.

• Align $d_{\text{model}}$ to hardware tile sizes. Always choose $d_{\text{model}}$ divisible by 128 at minimum; divisible by 256 is safer for H100 and future hardware. Matmul kernels use tiles of 128 or 256 elements; misaligned dimensions waste tile slots and silently hurt throughput by 10-20%.

• Let your parallelism plan constrain the choices. If tensor-parallel degree is 8, $d_{\text{model}}$ must be divisible by 8 and attention head count must accommodate the split. If pipeline-parallel across 4 nodes, $n_{\text{layers}}$ must be divisible by 4 and ideally by 8 (to allow 2-way micro-batch interleaving within a stage).

• Wider favors tensor parallel; deeper favors pipeline parallel. Fast NVLink within a node ($\geq 600$ GB/s) enables tensor parallelism at scale. Slower inter-node InfiniBand (100-400 Gbps) makes tensor parallelism across nodes expensive. Your networking budget shapes the answer before model quality does.

• Downstream accuracy may mildly favor depth, but do not let it override systems constraints. The quality argument for going deeper is real but modest. If adding depth would shatter pipeline efficiency or require redesigning your cluster topology, the quality gain rarely justifies the operational cost.

How This Shows Up in Interviews

• “How would you choose $d_{\text{model}}$ and $n_{\text{layers}}$ for a 13B model?” Strong answer: apply the 128-per-layer rule to get $d_{\text{model}} = 5120$, $n_{\text{layers}} = 40$-this matches LLaMA-2 13B. Then note that both numbers need to be divisible by the intended tensor-parallel degree, and $n_{\text{layers}}$ must divide cleanly across pipeline stages.

• “Why does aspect ratio matter beyond model quality?” Strong answer: width determines tensor-parallel degree feasibility because you partition weight matrices along the hidden dimension; depth determines pipeline stage count and stage granularity; both interact with your cluster’s networking topology in ways that can dominate cost long before quality differences appear.

• “If pretraining loss is equal for a wide-shallow and a narrow-deep model, which do you ship?” Strong answer: consider downstream task performance (deeper may win), then evaluate which is cheaper to serve at inference. Narrower-but-deeper models have smaller KV cache per layer at a given $d_{\text{model}}$, which can reduce memory pressure during long-context generation; wider models may serve faster if tensor parallelism can saturate hardware.

• Senior follow-up: “Your cluster has NVLink within 8-GPU nodes but only 200 Gbps InfiniBand between nodes. You have a 70B parameter budget. Walk through your aspect ratio and parallelism plan.” Sketch: with 8-way NVLink per node, tensor-parallel degree 8 is viable within a node. With slow internode fabric, avoid tensor parallelism across nodes; use pipeline parallelism across nodes instead. Target $d_{\text{model}} = 8192$ (divisible by 8 for TP-8), $n_{\text{layers}} = 80$ (pipeline across 4 nodes of 20 layers each, with 8-way TP within each node). Aspect ratio $\approx 102$-inside the validated band. Total GPUs: $8 \times 4 = 32$ H100s, a plausible allocation for a 70B model.

Key Takeaways

• The loss-optimal aspect ratio $d_{\text{model}} / n_{\text{layers}}$ sits between 100 and 128 across all publicly studied scales; use 128 as your default and adjust only for systems reasons.

• This optimum appears stable from 50M to 175B$+$ parameters, meaning small-scale experiments can validate your choice before committing to a full run.

• Aspect ratio determines which parallelism strategies are feasible: wider models favor tensor parallelism (needs NVLink), deeper models favor pipeline parallelism (tolerates slower inter-node links).

• Always align $d_{\text{model}}$ to multiples of 128 or 256 to avoid wasted CUDA tile slots; hardware alignment is non-negotiable, not a nice-to-have.

• Downstream accuracy may mildly favor depth at equal pretraining loss, but the systems cost of going very deep is real and often overrides this preference in production.

4.1.2 Feed-Forward to Model Dimension Ratios

Every interviewer who asks you to spec out an LLM architecture will eventually ask: “what inner dimension do you use for the FFN?” Engineers who know only the $4 \times$ rule freeze the moment they switch to SwiGLU and realize the formula changes. The correct answer-and the eighty-second derivation behind it-immediately separates candidates who understand why the numbers are what they are from those who have memorized a config file. This question appears across both mid-level and staff-level interviews; at staff level the follow-up involves the parameter budget algebra.

A standard transformer FFN without bias terms is:

where $W_{1} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$ and $W_{2} \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$. Total parameter count per layer: $2 \cdot d_{\text{model}} \cdot d_{\text{ff}}$. The near-universal default is:

giving $8 \cdot d_{\text{model}}^{2}$ parameters per FFN layer. This is not derived from first principles-it is a convention that sits inside a wide, empirically flat loss basin and has become the community standard.

For GLU variants (SwiGLU, GeGLU), the operation introduces a gate projection:

$\text{up} \qquad \text{gate}^{\text{model} \qquad \text{glu} \text{ff}} \qquad \text{down}^{\text{glu}}$

where $W , W \in \mathbb{R}^{d \qquad \times d}$ and $W \qquad \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$. Three matrices instead of two, so the parameter count is $3 \cdot d_{\text{model}} \cdot d_{\text{ff}}^{\text{glu}}$ .

To keep the GLU parameter count equal to the vanilla $4 \times$ FFN:

The rule: $4 \times$ for vanilla (ReLU/GeLU) MLPs; $8 / 3 \approx 2 . 67 \times$ for GLU variants. The factor of $2 / 3$ neutralizes the cost of the third weight matrix so the two architectures are parameter-matched by default.

The $4 \times$ convention survives not because it is optimal, but because hyperparameter sweeps across multiple model scales show that the loss landscape is essentially flat from roughly $2 \times$ to $8 \times$. There is no quality signal that would prefer $4 \times$ over $3 . 5 \times$ or $5 \times$. Having a community standard avoids accidental parameter inflation when comparing architectures and makes configs and ablations interpretable.

The $8 / 3$ correction for GLU models exists precisely to prevent an accidental parameter increase. If you kept $d_{\text{ff}} = 4 \cdot d_{\text{model}}$ while switching to SwiGLU, you would add a third weight matrix and inflate FFN parameters by $50 \%$-giving the GLU model a hidden parameter advantage in any comparison with a vanilla MLP. The $8 / 3$ rule neutralizes this. LLaMA-1, Qwen, DeepSeek, and Yi all follow the $\approx 2 . 67 \times$ convention closely; LLaMA-2 and Mistral run slightly higher ($\approx 2 . 69 \times$ due to rounding) but the intent is the same.

T5’s 11B model is the instructive outlier: $d_{\text{model}} = 1024$ with $d_{\text{ff}} = 65 , 536$, a ratio of $64 \times$. The stated rationale was that very wide inner projections create larger, more hardware-efficient matrix multiplies. This is technically true-wider matmuls can better saturate tensor cores-but the model is spending its parameter budget in a single, highly parallel expansion rather than distributing it across depth, attention, and width. T5v1.1, the improved follow-up, reverted to a $\approx 2 . 5 \times$ GeGLU ratio and obtained a better model. That reversion is the clearest available evidence that extreme expansion ratios do not pay off.

A Worked Example

Design the FFN for a 7B-parameter dense model using SwiGLU with $d_{\text{model}} = 4096$.

Step 1: compute the target inner dimension.

Step 2: round to hardware-friendly granularity. CUDA matmul kernels use tiles of 64 or 128 elements; the inner dimension must be a multiple of 64 at minimum. The two nearest multiples of 64 are 10880 and 10944; the nearest multiple of 128 above the target is 11008. LLaMA-2 7B uses 11008.

Step 3: verify parameter matching.

Vanilla $4 \times$ MLP at $d_{\text{model}} = 4096$:

SwiGLU at $d^{\text{glu}} = 11008$:

$\text{ff}$

The $\approx 1 \%$ discrepancy comes from rounding to the nearest multiple of 64. Across 32 layers, the total FFN delta is $\approx 33$M parameters out of $\approx 7$B-negligible and expected. This is why you will see minor parameter count discrepancies in model cards when comparing vanilla and GLU architectures of nominally the same size.

Step 4: check tensor-parallel compatibility. With 2-way TP, $11008 / 2 = 5504$, which is divisible by 64. With 4-way TP, $11008 / 4 = 2752$, still divisible by 64. The dimension survives any reasonable tensor-parallel degree.

What You Actually Decide in Practice

• Always round $d_{\text{ff}}$ to a multiple of 128. A multiple of 64 is the minimum; 128 is safer across H100 and is required for some fused kernel implementations. Misaligned inner dimensions silently reduce matmul throughput by 10-20% because tile slots are wasted.

• Use $4 \times$ for ReLU/GeLU; use $8 / 3 \times$ for SwiGLU/GeGLU. Applying $4 \times$ to a SwiGLU model inflates FFN parameters by $50 \%$ and breaks any fair comparison with vanilla MLP architectures. This error is common when porting GPT-style configs to LLaMA-style architectures.

• The range $2 \times$ to $8 \times$ is essentially flat in pretraining loss. You have real flexibility inside this band. Adjusting $d_{\text{ff}}$ is a clean lever to hit a specific total parameter count without touching $d_{\text{model}}$, $n_{\text{layers}}$, or attention configuration.

• Ratios above $16 \times$ carry quality risk. T5’s $64 \times$ experiment worked but T5v1.1 reverted to $2 . 5 \times$ with a better outcome. Gemma 2 uses $8 \times$ as a deliberate architectural choice, which is the most aggressive ratio with clear production validation. Beyond $8 \times$, you are outside the range where multiple independent experiments have validated quality.

• In MoE models, $d_{\text{ff}}$ is per expert. Track both the per-expert FFN dimension and the activated parameter count separately from total parameters. Applying the $8 / 3$ rule to the per-expert inner dimension is correct; do not accidentally apply it to the combined expert capacity.

How This Shows Up in Interviews

• “You are designing a 7B model with SwiGLU. What inner FFN dimension do you use?” Strong answer: compute $8 / 3 \times 4096 \approx 10923$, round to 11008 (nearest multiple of 128). Point out this matches LLaMA-2 7B exactly. Explain that keeping $4 \times$ with SwiGLU would silently add $50 \%$ more FFN parameters-an apples-to-oranges comparison.

• “Why is the FFN expansion ratio $4 \times$ instead of $2 \times$ or $8 \times$?” Strong answer: $4 \times$ sits in the middle of a wide, empirically flat loss basin that spans roughly $2 \times$ to $8 \times$. There is no strong quality signal distinguishing values in that range. The community adopted $4 \times$ as a focal point; having a standard makes configs interpretable and ablations comparable across papers.

• “What is the FFN dimension for a SwiGLU model if $d_{\text{model}} = 8192$?” Strong answer: $8 / 3 \times 8192 = 21845 . 3$; round to 21888 (multiple of 128) or 22016 (another clean multiple). The final choice depends on tensor-parallel degree: if TP is 8, confirm $21888 / 8 = 2736$ is divisible by 64.

• Senior follow-up: “With a fixed 7B parameter budget, if you increase $d_{\text{ff}}$ from 11008 to 16384, what else must change to keep total parameters constant?” Sketch: FFN parameters per layer increase by $( 16384 - 11008 ) \times 3 \times 4096 \approx 66$M. Across 32 layers that is $\approx 2 . 1$B extra parameters. To stay at 7B, you must cut elsewhere-either reduce $n_{\text{layers}}$ (which destroys the aspect ratio) or reduce $d_{\text{model}}$ to $\approx 3584$ (which changes all attention head dimensions and the embedding size). The cascade shows why FFN dimension is not an isolated knob; it couples to every other hyperparameter through the total parameter budget.

Key Takeaways

• Use $d_{\text{ff}} = 4 \cdot d_{\text{model}}$ for ReLU/GeLU MLPs; use $d_{\text{ff}} = ( 8 / 3 ) \cdot d_{\text{model}}$ for SwiGLU/GeGLU-the factor of $2 / 3$ preserves parameter count by accounting for the third (gate) weight matrix.

• Always round $d_{\text{ff}}$ to a multiple of 128 for CUDA tile alignment; misalignment silently costs 10-20% matmul throughput.

• The loss landscape is flat between $2 \times$ and $8 \times$, giving genuine flexibility; values outside that range need strong justification.

• T5’s $64 \times$ ratio is the cautionary tale: T5v1.1 reverted to $\approx 2 . 5 \times$ GeGLU and improved; extremely wide FFNs trade expressiveness for hardware efficiency in a way that rarely pays off at inference.

• In MoE architectures, $d_{\text{ff}}$ is per expert; always separate per-expert FFN parameters from total model parameters when reporting model capacity.

4.2 Attention Head Configuration

4.2.1 Head Count vs. Head Dimension Trade-Offs

“Design attention for a 13B model with $d_{\text{model}} = 5120$” is a standard architecture deep-dive prompt. The answer-40 heads at 128 dimensions each-resolves instantly once you internalize the governing constraint. What trips engineers is the follow-up: why this ratio, what breaks when you deviate from it, and how does the choice propagate into tensor-parallelism planning and inference memory? This question appears at mid-level and stafflevel interviews alike; the staff variant adds explicit KV cache sizing and the implications of grouped-query attention.

In standard multi-head attention, queries, keys, and values are each projected via weight matrices $W_{Q} , W_{K} , W_{V} \in \mathbb{R}^{d_{\text{model}} \times ( n_{\text{heads}} \cdot d_{\text{head}} )}$. The near-universal constraint is:

This makes each projection effectively square in aggregate, and the total attention parameter count-four matrices $W_{Q}$, $W_{K}$, $W_{V}$ , $W_{O}$-stays fixed at $4 \times d_{\text{model}}^{2}$ per layer regardless of how many heads you use. Adding heads subdivides the existing projection space; it does not expand it.

The per-head dimension follows directly:

$\text{heads}$ For the three LLaMA-2 sizes: a 7B model with $d_{\text{model}} = 4096$ and $n_{\text{heads}} = 32$ gives $d_{\text{head}} = 128$; a 13B model with $d_{\text{model}} = 5120$ and $n_{\text{heads}} = 40$ gives $d_{\text{head}} = 128$; a 70B model with $d_{\text{model}} = 8192$ and $n_{\text{heads}} = 64$ gives $d_{\text{head}} = 128$. The pattern is not coincidence. When aspect ratio is pinned near 128 ($d_{\text{model}} / n_{\text{layers}} \approx 128$), fixing $d_{\text{head}} = 128$ and applying the constraint above mechanically produces the correct $n_{\text{heads}}$ at every model scale.

GPT-3, LaMDA, PaLM, and all publicly documented LLaMA-family models follow the ratio-1 constraint. T5 11B is the only notable deviation: with $d_{\text{model}} = 1024$, $n_{\text{heads}} = 128$, and $d_{\text{head}} = 128$, the total attention projection is $128 \times 128 = 16 , 384 = 16 \times d_{\text{model}}$. The model trained, but T5v1.1 reverted to the standard ratio and achieved better perplexity.

Pin $d_{\text{head}} = 128$, derive $n_{\text{heads}} = d_{\text{model}} / 128$, and you will match every production configuration that has been publicly validated; departures from this rule require explicit justification in terms of quality or systems trade-offs.

Multiple heads give attention the ability to express diverse patterns simultaneously. One head might track adjacent-token dependencies, another follow long-range coreference, a third attend to syntactic head-modifier relationships. That diversity is the core mechanism through which multi-head attention improves over single-head attention. But each head’s output is a weighted combination of value vectors in a $d_{\text{head}}$-dimensional subspace. The smaller $d_{\text{head}}$ is, the lower the rank of what any individual head can represent.

This is the low-rank bottleneck: with very small $d_{\text{head}}$ (below roughly 64), the per-head attention output is intrinsically rank-limited, and the head cannot represent nuanced multi-directional aggregations even if the attention weights are well-calibrated. In practice, $d_{\text{head}} = 64$ is the minimum seen in any well-performing production architecture; 128 is the empirically validated sweet spot across all publicly studied model scales.

The opposing failure mode is too few, very wide heads. With $n_{\text{heads}} = 8$ and $d_{\text{head}} = 512$ at $d_{\text{model}} = 4096$, each head is highly expressive but there are only eight distinct attention patterns to aggregate. In practice the diversity loss dominates: fewer heads means fewer independent signals for the residual stream to combine, and the model underperforms a 32-head configuration at identical parameter count. The empirical sweet spot is $n_{\text{heads}}$ large enough that heads genuinely specialize-typically 16 or more at production scale-with $d_{\text{head}}$ large enough to avoid rank bottlenecks.

A Worked Example

Design attention for a 13B model: $d_{\text{model}} = 5120$, $n_{\text{layers}} = 40$.

Step 1: apply the constraint.

Step 2: verify total attention parameters per layer.

Step 3: check two alternatives.

$n_{\text{heads}} = 80$, $d_{\text{head}} = 64$: total attention parameters are identical-same constraint-but each head now operates in a 64-dimensional space at the low-rank boundary. Viable, but at the limit of the validated range.

$n_{\text{heads}} = 20$, $d_{\text{head}} = 256$: same parameter count, but only 20 distinct attention patterns. Diversity loss typically outweighs the per-head expressiveness gain at this scale.

Step 4: verify tensor-parallel divisibility. With TP degree 8: $40 / 8 = 5$ heads per GPU shard-clean. $d_{\text{head}} = 128$ is a multiple of 64, satisfying CUDA tile alignment. Both constraints pass for the standard choice. As a contrast: if $d_{\text{model}} = 5376$ were chosen, $n_{\text{heads}} = 5376 / 128 = 42$, and $42 / 8 = 5 . 25$-not clean, requiring a revision to $d_{\text{model}}$ before locking the architecture.

Step 5: estimate the MHA KV cache per token per layer (bf16).

At 4096 tokens and batch size 8 across 40 layers:

This is the MHA KV cache baseline that motivates grouped-query attention at larger scale.

What You Actually Decide in Practice

• Pin $d_{\text{head}} = 128$ and derive $n_{\text{heads}}$. Every production model between 7B and 175B with publicly documented architecture uses $d_{\text{head}} = 128$. Start here, compute $n_{\text{heads}} = d_{\text{model}} / 128$, and round only if forced to by divisibility constraints.

• Enforce the ratio-1 constraint: $n_{\text{heads}} \times d_{\text{head}} = d_{\text{model}}$. Violating this changes total attention parameter count and creates non-comparability with standard baselines. If you expand attention projection beyond $d_{\text{model}}$ (as T5 11B did), count those extra parameters explicitly in your model card-they are not free.

• Make $n_{\text{heads}}$ divisible by TP degree before finalizing $d_{\text{model}}$. If tensor-parallel degree is 8, $n_{\text{heads}}$ must be a multiple of 8. This constraint, combined with the ratio-1 rule and the aspect-ratio target, leaves only a discrete set of valid $d_{\text{model}}$ values. Enumerate these early-discovering a conflict after the training run has started is expensive.

• Do not go below $d_{\text{head}} = 64$. Models with $d_{\text{head}} = 32$ or $d_{\text{head}} = 16$ show measurable quality penalties from low-rank per-head bottlenecks. GPT-2 small ($d_{\text{head}} = 64$) represents the minimum with welldocumented strong performance; anything below is outside the validated range.

• The MHA KV cache per token is $4 \times d_{\text{model}}$ bytes. This fixed per-token cost is the direct motivation for grouped-query attention. For a 70B model at $d_{\text{model}} = 8192$, MHA costs 32 KB per token per layer; at 100K context and batch 16 across 80 layers, that is 4.2 TB-clearly infeasible. GQA cuts this by a factor of the group size $G$.

How This Shows Up in Interviews

• “For a 7B model with $d_{\text{model}} = 4096$, how many attention heads do you use and why?” Strong answer: $n_{\text{heads}} = 32$, giving $d_{\text{head}} = 128$. This satisfies $n_{\text{heads}} \times d_{\text{head}} = d_{\text{model}}$, keeps total attention parameters at $4 \times 4096^{2}$ per layer, and hits the empirically validated $d_{\text{head}} = 128$ target. Point out this is exactly LLaMA-2 7B’s configuration, which validates the reasoning.

• “What happens if you double the number of attention heads to 64 for a 7B model?” Strong answer: $d_{\text{head}}$ drops from 128 to 64-at the low-rank boundary. Total attention parameters are unchanged. Diversity of patterns increases, but per-head expressiveness drops. In practice, no frontier model at 7B scale has found benefit from this change; the additional diversity does not compensate for the reduced per-head capacity.

• Senior follow-up: “Your cluster uses TP-8. You want a 70B model with $d_{\text{model}} = 8192$. Walk through your head count and verify the tensor-parallel constraint, including for GQA.” Sketch: target $d_{\text{head}} = 128$, so $n_{\text{heads}} = 8192 / 128 = 64$. TP-8 divisibility: $64 / 8 = 8$ heads per GPU shard-clean. For GQA-8 ($n_{\text{kv}} = 8$): $8 / 8 = 1$ KV head per shard-clean, just barely. For GQA-4 ($n_{\text{kv}} = 16$): $16 / 8 = 2$-also fine. The dangerous case is $n_{\text{kv}} = 6$: $6 / 8$ is not an integer, requiring either a different TP degree or a different GQA configuration.

Key Takeaways

• The universal constraint is $n_{\text{heads}} \times d_{\text{head}} = d_{\text{model}}$; this fixes total attention parameters at $4 \times d_{\text{model}}^{2}$ per layer regardless of head count.

• Target $d_{\text{head}} = 128$ and derive $n_{\text{heads}}$ from it; this matches GPT-3, PaLM, and all LLaMA-family models at every scale.

• Avoid $d_{\text{head}} < 64$; the low-rank per-head bottleneck produces measurable quality degradation below this threshold.

• $n_{\text{heads}}$ must be divisible by tensor-parallel degree; verify this before finalizing $d_{\text{model}}$, as the three constraints (aspect ratio, ratio-1, TP divisibility) interact.

• MHA KV cache per token per layer is $4 \times d_{\text{model}}$ bytes (bf16); this fixed cost per-token is what makes grouped-query attention necessary at production scale and long contexts.

4.2.2 Consensus Choices Across Frontier Models

Multi-head attention is the right choice during training. The problem surfaces when your model enters production: at batch size 32 and 8K context, the KV cache for a 70B MHA model can exceed the aggregate HBM capacity of a four-H100 serving node. Grouped-query attention (GQA) is how frontier labs have resolved this without sacrificing model quality, and it is now a prerequisite in any serious inference system design discussion. This question appears at senior and staff level as part of a serving architecture deep-dive, and the expected answer goes beyond naming GQA to computing the KV cache size and justifying the group count.

Three variants of multi-head attention differ only in how many distinct key-value heads exist:

• Multi-head attention (MHA): $n_{\text{kv}} = n_{q}$. Each query head has its own $W_{K}$ and $W_{V}$ . Full representational capacity. Full KV cache per token.

• Multi-query attention (MQA): $n_{\text{kv}} = 1$. All query heads share a single $W_{K}$ and $W_{V}$ . KV cache reduced by factor $n_{q}$. Shows measurable quality loss at large model scale.

• Grouped-query attention (GQA): $n_{\text{kv}} = n_{q} / G$ for group size $G$. Groups of $G$ query heads share $W_{K}$ and $W_{V}$ . KV cache reduced by factor $G$. Quality loss is empirically negligible for $G \leq 8$.

The KV cache per token per layer in bf16 is:

With the ratio-1 constraint ($n_{q} \times d_{\text{head}} = d_{\text{model}}$), this becomes:

GQA directly scales down the KV cache by the group factor $G$ with no change to the query projection.

GQA at $G = 4$-$8$ is the current frontier consensus: it reduces KV cache by $4$-$8 \times$ with quality degradation that is empirically indistinguishable from MHA on standard benchmarks.

The key intuition is that query diversity matters more than key-value diversity. Queries express what each head is searching for; they need to be distinct to capture different aspects of the context simultaneously. Key and value representations encode the information being retrieved-multiple query heads can retrieve from the same K/V pool effectively because the queries direct where to look. GQA preserves the full query projection ($n_{q}$ distinct $W_{Q}$ matrices) while sharing the representation being queried.

The memory-quality Pareto is well-characterized. MQA-sharing a single KV head across all query headssaves the most memory but causes measurable quality degradation at large model scale: having one KV head for 64 or more query heads is too aggressive a compression of the context representation. GQA at $G = 4$ to $G = 8$ matches MHA on perplexity and downstream benchmarks when trained from scratch. GQA at $G \geq 16$ begins to show degradation comparable to MQA for models above 30B parameters. The sweet spot is $G = 8$ for 70B-class models; $G = 4$ is increasingly common for 7B-13B models.

The memory reduction is not just a convenience-it is what enables longer context windows and larger serving batches. For a fixed HBM budget, GQA-8 allows either $8 \times$ more tokens in the context, $8 \times$ more requests in a batch, or some combination of both. This has a direct and quantifiable effect on serving throughput.

A Worked Example

LLaMA-2 70B configuration: $d_{\text{model}} = 8192$, $n_{q} = 64$, $n_{\text{kv}} = 8$ (GQA-8), $d_{\text{head}} = 128$, $n_{\text{layers}} = 80$.

Step 1: KV cache per token per layer.

Step 2: total KV cache at serving parameters (context 4096, batch 8).

Step 3: compare with MHA ($n_{\text{kv}} = 64$).

At this configuration MHA KV cache alone equals or exceeds the entire 80 GB HBM of a single H100-before a single byte of model weights.

Step 4: model weight memory. 70B parameters in bf16: $70 \times 10^{9} \times 2 = 140$ GB. Across 4 H100s with TP-4: $140 / 4 = 35$ GB per GPU.

Step 5: total memory per GPU (GQA-8). Assuming KV cache is sharded by TP degree: $10 . 7 / 4 \approx 2 . 7$ GB per GPU for KV. Total: $35 + 2 . 7 = 37 . 7$ GB-well within the 80 GB limit, leaving 42 GB of headroom for activations, longer sequences, or increased batch size.

With MHA instead: $85 . 7 / 4 \approx 21 . 4$ GB per GPU KV cache. Total: $35 + 21 . 4 = 56 . 4$ GB per GPU-still fits, but leaves only $\sim$24 GB of headroom, making 8K context or batch sizes above 8 impractical. At context 8192 with MHA the KV cache doubles to 171 GB total; GQA-8 at 8192 context is $10 . 7 \times 2 = 21 . 4$ GB-still comfortable.

What You Actually Decide in Practice

• Use MHA for models $\leq 7$B serving short contexts. At 7B ($d_{\text{model}} = 4096$), the MHA KV cache per token is half that of a 70B model, and memory pressure is manageable for batch sizes under 16 and contexts under 4K. Once either dimension grows, switch to GQA-4.

• Default to GQA-8 for 13B-70B models; GQA-4 for 7B-13B. LLaMA-2 70B, LLaMA-3 70B, and most frontier 70B-class models use $n_{\text{kv}} = 8$. LLaMA-3 8B uses $n_{\text{kv}} = 8$ with $n_{q} = 32$ (also GQA-4). These represent the production-validated sweet spots on the memory-quality Pareto.

• Reserve MQA for extreme memory constraints. MQA’s quality penalty is real and documented at 70B$+$ scale. Use it only when even GQA-8 does not fit your HBM budget-for example, on inference chips with $\leq 8$ GB of memory or edge deployments.

• $n_{\text{kv}}$ must be divisible by tensor-parallel degree. This is the most common configuration error when adapting a GQA model to a new cluster. With TP-8, $n_{\text{kv}}$ must be $\geq 8$ and a multiple of 8. GQA-8 on a 64-head model gives $n_{\text{kv}} = 8$-exactly 1 KV head per shard at TP-8. GQA-4 on the same model gives $n_{\text{kv}} = 16$-2 per shard, also clean. GQA-4 on a 40-head model gives $n_{\text{kv}} = 10$-not divisible by 8, requiring either a different GQA group size or a different TP degree.

• GQA is a training-time decision, not an inference-time optimization. You cannot convert a trained MHA model to GQA at inference without retraining. The $W_{K}$ and $W_{V}$ weights are distinct per head in MHA; merging them post-hoc by averaging degrades quality measurably. If GQA is needed for your serving cluster, specify it before the training run starts.

How This Shows Up in Interviews

• “Why does LLaMA-2 70B use GQA instead of standard multi-head attention?” Strong answer: KV cache memory. At 4096 context and batch size 8, MHA requires $\sim$86 GB of KV cache for a 70B model-exceeding a single H100’s HBM and leaving no headroom for weights alongside the cache on any reasonable serving cluster. GQA-8 reduces this to $\sim$11 GB, fitting comfortably within a 4-GPU serving configuration. Quality on benchmark evaluations is statistically identical.

• “What is the difference between MQA and GQA-8, and when do you choose each?” Strong answer: both reduce KV cache by sharing $W_{K}$ and $W_{V}$ across query heads. MQA uses one KV head (maximum reduction, by factor $n_{q}$); GQA-8 uses $n_{q} / 8$ KV heads (reduction by factor 8). GQA-8 shows negligible quality loss at 70B scale; MQA shows measurable degradation for large models. Choose GQA-8 as the default; use MQA only when even GQA-8 exceeds your HBM budget.

• Senior follow-up: “You are serving a 70B GQA-8 model ($n_{q} = 64$, $n_{\text{kv}} = 8$, $d_{\text{head}} = 128$, 80 layers) at context 8192, batch 32, TP degree 4. Compute the KV cache per GPU and assess whether it fits on an H100.” Sketch: KV bytes per token per layer $= 2 \times 8 \times 128 \times 2 = 4 , 096$. Total KV cache $= 4 , 096 \times 8192 \times 32 \times 80 \approx 85 . 9$ GB. Sharded by TP-4: $85 . 9 / 4 \approx 21 . 5$ GB per GPU. Model weights: $140 / 4 = 35$ GB per GPU. Total: $35 + 21 . 5 = 56 . 5$ GB-fits within 80 GB with $\sim$23 GB headroom for activations. The configuration is viable.

Key Takeaways

• MHA, MQA, and GQA differ by $n_{\text{kv}}$: equal to $n_{q}$ (MHA), 1 (MQA), or $n_{q} / G$ (GQA); KV cache scales linearly with $n_{\text{kv}}$ and is the dominant memory constraint during inference.

• GQA at $G = 4$-$8$ is the frontier consensus: $4$-$8 \times$ KV cache reduction with quality loss empirically indistinguishable from MHA; MQA causes measurable degradation at 70B$+$ scale.

• $n_{\text{kv}}$ must be divisible by TP degree; this is the single most common configuration error when adapting a GQA architecture to a new cluster topology.

• GQA must be specified at training time-post-hoc MHA-to-GQA conversion degrades quality and is not a standard deployment practice.

• Size your serving cluster by computing MHA KV cache first ($4 \times d_{\text{model}} \times$ seq $\times$ batch $\times n_{\text{layers}}$ bytes in bf16), then dividing by $G$; leave 20-30% headroom for activations and metadata.

4.3 Vocabulary Size and Tokenization Impact

4.3.1 Monolingual vs. Multilingual Vocab Sizing

An interviewer asks you to justify your tokenizer vocabulary size. You say 32,768. They follow up: the model will serve 50 languages and handle code. The number sounds embarrassingly small. Vocabulary size is one of those hyperparameters that feels like an implementation detail until you realize it directly controls sequence length, embedding matrix cost, and whether speakers of minority languages pay a three-to-four times token tax compared to English speakers. This question appears at every seniority level, but the depth of expected answer scales sharply with the role.

The vocabulary is the set of distinct token identifiers the model recognizes. A BPE tokenizer builds this set by starting from raw bytes (256 elements) and greedily merging the most frequent adjacent pair into a new token, repeating until a target vocabulary size is reached. Each merge compresses the corpus: two common adjacent tokens become one. The vocabulary size $V$ is therefore both a compression budget and a coverage budget simultaneously.

The compression quality of a tokenizer is captured by its compression ratio: the average number of bytes encoded per token. If a tokenizer achieves a ratio of $c$ on a given corpus, then a document of $B$ bytes produces $L = B / c$ tokens. For GPT-2’s 50,257-token BPE vocabulary trained on English web text, this ratio is roughly 1.6 bytes per token on English prose. Byte-level tokenization achieves exactly 1.0 by definition, which forces sequences to be four to eight times longer than BPE for the same text-a prohibitive cost for any attention-based model.

A larger vocabulary lets BPE find and merge higher-frequency multi-character patterns, raising the compression ratio. A smaller vocabulary forces rare character sequences to survive as many short tokens, inflating sequence lengths. The cost side: each extra vocabulary element adds one row to the input embedding matrix and one row to the output unembedding matrix, contributing $2 \times V \times d_{\text{model}}$ parameters total. For a model with $d_{\text{model}} = 4096$, moving from $V = 32 , 768$ to $V = 131 , 072$ adds roughly 800 million parameters purely in the embedding tables-parameters that are accessed sparsely and do not dominate training FLOPs.

The vocabulary size is a compression-versus-embedding-cost dial: a larger $V$ shortens sequences at the price of a heavier embedding matrix, and the right setting shifts dramatically once you add languages beyond English.

Monolingual English models settled around 30,000-50,000 tokens (GPT-2 at 50,257; early LLaMA-1 at 32,000) because English text at those sizes achieves compression ratios comfortably above 1.5 and the embedding overhead stays manageable. Once you add non-Latin scripts-Arabic, Chinese, Korean, Thai, Hindi-the situation breaks. Those scripts contain characters underrepresented in a 50k vocabulary trained on Englishdominant web text. A single CJK character that appears infrequently in the tokenizer training corpus may not survive to become its own merge and must instead be encoded as three or four byte-level fragments. This fertility problem means a 50-character Chinese sentence may tokenize to 150-200 tokens while its English translation of 60 characters tokenizes to 70. The model runs an invisible length tax on non-English users.

The resolution is to train the tokenizer on a multilingual corpus with controlled per-language sampling, then expand $V$ to 100,000-250,000. The extra merge budget provides enough operations to build high-frequency multi-character tokens for each language. The compression ratio for previously underserved languages improves substantially: multilingual-aware tokenizers can reduce token counts for many non-English languages by 30- 60% compared to an English-only 50k vocabulary at the same model size. The embedding overhead is real-roughly 800M to 1.6B extra parameters for a 4k-7k hidden dimension at 128k-256k vocab-but those parameters are cheap: sparse access means they never dominate compute.

A secondary benefit of a larger vocabulary is reduced sequence length for code. Tokenizers that recognize programming language keywords, operators, and common identifiers as single tokens cut token counts for source files by 20-40% versus generic text tokenizers, which matters when processing long files near context limits.

A Worked Example

Consider a multilingual assistant with $d_{\text{model}} = 4096$ and 32 transformer layers.

Option A: 50,000-token English-centric BPE vocabulary.

• Embedding and unembedding parameter cost: $2 \times 50 , 000 \times 4096 \approx 410$M parameters.

• A 128-character Hindi sentence tokenizes to roughly 160-200 tokens (many Devanagari characters split into byte pairs).

• At a context budget of 2,048 tokens, practical Hindi context is limited to around 1,300 characters.

• Compression ratio on Hindi: approximately 0.6-0.8 bytes per token-well below 1.0, meaning bytes per token is less than one because multibyte characters get split.

Option B: 128,000-token multilingual BPE vocabulary.

• Embedding and unembedding cost: $2 \times 128 , 000 \times 4096 \approx 1 , 048$M parameters-an increase of roughly 640M parameters, or about 1.2 GB in bf16.

• The same 128-character Hindi sentence tokenizes to roughly 60-80 tokens, a 2.5x improvement.

• Under the same 2,048-token budget, Hindi context expands to around 3,300 characters.

• English compression ratio is largely unchanged at $\sim$1.6 bytes/token.

The embedding table grows by 640M parameters-negligible relative to the 28B+ non-embedding parameters in a typical production model of this size. The gain in usable context for underserved languages is substantial and directly observable in downstream task quality on non-English benchmarks.

What You Actually Decide in Practice

• English-only or narrow-domain model: 32k-64k tokens is sufficient. Early LLaMA at 32k, GPT-2 at 50k, and Mistral 7B at 32k all demonstrate strong English performance at these sizes. Adding more vocabulary for a monolingual English use case yields diminishing returns past 64k.

• Multilingual or production assistant: Target 100k-200k. GPT-4’s tiktoken vocabulary is 100,277; LLaMA-3 expanded to 128,256; Qwen2 uses 151,936. These numbers reflect the point where per-language compression ratios plateau for the major world languages plus code.

• Tokenizer training corpus balance matters as much as vocabulary size: A 128k vocabulary trained on 95% English data will still underserve minority languages. When training the tokenizer, oversample rare languages-if your pre-training corpus is 80% English and 20% other, train the tokenizer on a corpus that is 60% English and 40% other to force more merge budget toward non-English character patterns.

• Embedding parameter cost is rarely the bottleneck: A 128k vocabulary at $d = 4096$ costs about 1 GB in bf16. A 70B parameter model has roughly 130 GB of total weights. The embedding fraction is under 1%. Do not sacrifice multilingual coverage to save embedding memory.

• The vocabulary is frozen after pre-training: This is the most expensive mistake to fix. If you launch with a 32k vocabulary and later discover poor Korean performance, you cannot surgically add Korean tokens without retraining. Vocabulary extension via continued pre-training is imperfect and requires substantial additional compute. Pick the right vocabulary size before the first GPU-hour of pre-training runs.

How This Shows Up in Interviews

• “Why did you pick that vocabulary size, and how did you decide?” A strong answer names the target language mix, the compression ratio measured on held-out samples per language, the embedding parameter budget, and the irreversibility of the decision. Weak answers pick 32k because “that is what LLaMA used.”

• “If I tell you the model will serve users in 40 languages, what changes?” Cover the fertility problem for non-Latin scripts, the need to train the tokenizer on a balanced multilingual corpus, and the target range of 100k-200k tokens. Note that sampling strategy for tokenizer training matters: training on an English-dominant corpus defeats the purpose of a larger vocabulary.

• Senior follow-up: “We are doing continued pre-training to improve Hindi on an existing 32k-vocab model. Walk me through the risks.” The strong answer addresses: (1) vocabulary extension requires adding new embedding rows initialized randomly while existing rows carry trained representations-new and old parameters need separate learning rate treatment; (2) the model’s output distribution encodes assumptions about token probability that conflict with the new vocabulary’s statistics; (3) extended continued pre-training is required before the new tokens contribute meaningfully; and (4) the correct engineering decision is to train from scratch with the right vocabulary upfront, and vocabulary extension is an expensive compromise accepted only under strong resource constraints.

Key Takeaways

• Monolingual models: 32k-64k tokens. Multilingual or production systems: 100k-200k tokens. The jump is driven by the fertility problem in non-Latin scripts, where English-centric vocabularies produce 2-4x more tokens per character.

• A larger vocabulary reduces sequence length for underserved languages by 2-3x, directly expanding usable context and reducing inference cost per query.

• Embedding parameter overhead from 50k to 128k at $d = 4096$ is roughly 640M parameters ($\approx$1.2 GB in bf16)-under 1% of total weight in a 70B model and not a meaningful constraint.

• Tokenizer training corpus balance matters as much as vocabulary size: a large vocabulary trained on English-dominant data will still underserve minority languages.

• The vocabulary is frozen at pre-training. There is no cheap fix after the fact.

4.3.2 Tokenizer Selection and Downstream Cost

A product manager asks why API costs for Japanese users are running three times higher than for English users, and your engineering lead cannot explain it in the next five minutes. This is a tokenization fertility problem, and understanding it completely requires tracing the path from raw bytes to billed tokens. Tokenizer selection is also one of the few decisions that constrains every downstream system: sequence length budgets, KV-cache memory, training FLOPs, and per-query billing all live downstream of the tokenizer. The inability to swap tokenizers after pre-training makes this decision unusually high-stakes for an apparently low-level component.

Every token costs compute. At training time, the forward-backward pass cost scales roughly linearly with sequence length (ignoring the quadratic attention term). At inference time, prefill latency, decode cost, and KV-cache memory all scale with token count. Hosted API pricing is quoted per token, so the tokenizer’s compression quality directly determines the dollar cost of a given piece of text.

The key measurement is the fertility of a tokenizer on a specific text domain: the average number of tokens produced per unit of input (per word, per character, or per byte). Low fertility means high compression-fewer tokens, lower cost, longer effective context window. Fertility is not a fixed property of the tokenizer; it is a joint property of the tokenizer and the input distribution. A tokenizer trained on English web text will have low fertility on English and high fertility on Arabic or Thai.

Formally, if a tokenizer achieves a compression ratio $c$ bytes per token on a given domain, a document of $B$ bytes produces

tokens. Since inference cost scales as $\mathcal{O} ( L )$ for prefill and $\mathcal{O} ( L^{2} )$ for full attention over a sequence, the compression ratio has a multiplicative effect on total compute. A tokenizer with $c = 2 . 0$ bytes per token generates 50% fewer tokens than one with $c = 1 . 0$ on the same input-cutting prefill cost in half and allowing twice as much text to fit in the same context window.

The tokenizer is a silent cost multiplier: every downstream metric-sequence length, memory footprint, training FLOPs, inference billing-scales with the inverse of the compression ratio on the target domain.

BPE remains the dominant tokenization algorithm at production scale not because it is theoretically optimal but because it is simple, invertible, fast, and empirically effective. Its adaptive nature-merging pairs from the actual training corpus-means the vocabulary becomes a compact statistical summary of whatever text it was trained on. This is exactly the right property: a tokenizer that does maximum compression work per token on the specific data distribution it will encounter.

The failure mode is distribution mismatch between the tokenizer training corpus and the inference domain. An English-centric BPE tokenizer processes Arabic as a stream of bytes or very short byte-pair fragments because Arabic characters appear rarely enough in an English corpus that they never accumulate merge frequency. For a 100-character Arabic sentence, an English-only 50k BPE tokenizer might produce 150 tokens while a properly trained multilingual tokenizer produces 40-a 3.75x cost difference on that query alone.

A secondary cost that often goes unnoticed is the unembedding projection at inference time. For each generated output token, the model must compute a full matrix-vector product against the unembedding matrix of shape $V \times d_{\text{model}}$. At $V = 128 , 000$ and $d_{\text{model}} = 4096$ in bf16, this matrix occupies roughly 1 GB of memory and must be accessed on every decode step. In small models (sub-1B parameters), the unembedding projection can represent 30-50% of the total parameter count and becomes a meaningful memory-bandwidth bottleneck during autoregressive decode.

Alternatives to BPE-character-level, byte-level, unigram language model-have been studied but have not displaced BPE at frontier scale. Byte-level models eliminate the fertility problem entirely but produce sequences four to eight times longer than BPE for typical English text, making quadratic attention costs prohibitive. Unigram LM tokenizers (used by SentencePiece) achieve similar compression ratios to BPE and are more principled statistically, but have not demonstrated consistent downstream quality advantages. Until architectures that process bytes efficiently at frontier scale are proven, BPE variants remain the practical default.

A Worked Example

You are building an API for legal document summarization. Documents average 4,000 bytes of English legal prose. Evaluate two tokenizer options.

Option A: Generic English BPE, $V = 50 , 257$ (GPT-2 style)

• Compression ratio on legal text: $c \approx 1 . 4$ bytes/token. Legal text uses many long Latin-derived words (“indemnification,” “notwithstanding,” “hereinafter”) that appear rarely enough in the tokenizer training corpus to not have been fully merged.

• Tokens per document: $4 , 000 / 1 . 4 \approx 2 , 857$ tokens.

• At a hosted model price of $2.00 per million input tokens: cost per document $\approx \$ 0 . 0057$.

Option B: Domain-adapted BPE trained on legal corpora, $V = 64 , 000$

• Compression ratio on legal text: $c \approx 2 . 0$ bytes/token. Frequent legal phrases now appear as single tokens.

• Tokens per document: $4 , 000 / 2 . 0 = 2 , 000$ tokens.

• Cost per document: $\approx \$ 0 . 0040$.

• Savings: 30% lower per-document cost, and 30% more legal content fits in the same context window.

The tradeoff is that Option B requires training a custom tokenizer and pre-training or continued pre-training of the model with the new vocabulary. Option A allows immediate use of an existing foundation model. The domain-specific tokenizer makes economic sense only when document volume is large enough to amortize the re-training cost. At a million documents per day, the cost difference is $\$ 1 , 700$/day-a breakeven point reached quickly at scale.

What You Actually Decide in Practice

• Reusing an existing tokenizer: If you are fine-tuning or adapting a pretrained model, use its tokenizer without modification. Changing the vocabulary invalidates the entire embedding and unembedding matrix and forces a full pre-training run. The cost of tokenizer-domain mismatch-higher fertility, more tokens-is almost always lower than the cost of a wasted pre-training run.

• Training a new tokenizer: Justified only when building a new model from scratch. Train BPE on a representative sample of the target corpus-200M to 1B characters is typically sufficient for merge statistics to stabilize. Oversample underrepresented domains and languages relative to their share in the pre-training corpus: a 70/30 English/other split in the tokenizer training data when the pre-training corpus is 90/10 gives minority languages better coverage without hurting English compression quality materially.

• Measure fertility before committing: Before finalizing vocabulary size and training the tokenizer, compute compression ratios on held-out samples from every domain and language you care about. Any domain producing a ratio below 1.2 bytes/token is a red flag-fertility will be very high and costs will be disproportionately elevated for users in that domain. Fix the tokenizer, not the model.

• Context window arithmetic is token arithmetic: When users report fitting fewer documents than expected in the context window, check the tokenizer fertility on their specific input type. A model with a 128k-token context window effectively provides a much shorter context for high-fertility text such as dense mathematical notation, structured data, or low-resource language scripts.

• Embedding weight tying: Many models tie the input embedding matrix and the output unembedding matrix (they share the same weights), saving $V \times d_{\text{model}}$ parameters at no measured quality cost. At $V = 128 , 000$ and $d = 4096$, that is 524M parameters-about 1 GB in bf16. Always verify whether weight tying is used before reporting total parameter counts; the difference between tied and untied vocabularies at this size is significant.

How This Shows Up in Interviews

• “Why are our API costs higher for Japanese users than for English users?” A strong answer immediately names the tokenizer fertility problem: Japanese text produces more tokens per character under an English-trained vocabulary because Japanese character patterns were underrepresented in the tokenizer training corpus and did not accumulate enough merge frequency. The answer quantifies the expected multiplier (2-4x for CJK versus English) and outlines the fix: measure compression ratios per language, and if the model is being rebuilt, address vocabulary balance in the new tokenizer training data.

• “Can we retrain the tokenizer and swap it into our existing model?” No, not without full re-pretraining. The embedding and unembedding matrices are indexed by token ID; a new vocabulary means new IDs, which means the embedding rows no longer correspond to the trained representations. A strong answer explains this clearly and then discusses the cost of alternatives: continuing pre-training with vocabulary extension (imperfect, expensive), serving a separate model for high-fertility languages, or accepting the cost differential.

• Senior follow-up: “Your team is proposing a byte-level model to eliminate tokenization entirely. What would you need to see before committing to that architecture at production scale?” The senior answer names three things: (1) a credible solution to the 4-8x sequence length inflation-either subquadratic attention, very efficient sliding window attention, or a hierarchical architecture that first groups bytes before processing; (2) empirical evidence at scale that byte-level models match BPE models on standard benchmarks, which has not yet been demonstrated at frontier sizes; and (3) a plan for the inference serving infrastructure, since the KV-cache size scales with sequence length and a 5x sequence length increase dramatically changes memory requirements per concurrent request.

Key Takeaways

• Tokenizer compression ratio is a direct multiplier on inference cost, training FLOPs, and effective context length. A domain mismatch that cuts fertility by 2x doubles your token count and your cost on that domain.

• BPE fertility is not a fixed property of the tokenizer-it is a function of how well the tokenizer training corpus matched the inference domain. Measure fertility on held-out samples before finalizing vocabulary choices.

• Never change the tokenizer of an existing pretrained model. The embedding and unembedding matrices are tied to token IDs; a vocabulary swap requires full re-pretraining from scratch.

• Embedding weight tying (sharing input embedding and output unembedding weights) saves $V \times d_{\text{model}}$ parameters at no quality cost-roughly 1 GB at $V = 128 k$, $d = 4096$ in bf16. Verify whether it is used before counting parameters.

• Byte-level tokenization is architecturally elegant but produces 4-8x longer sequences than BPE for typical text; it has not been proven viable at frontier scale and requires either subquadratic attention or hierarchical architectures to become practical.

4.4 Regularization in Pre-Training

4.4.1 Why Dropout Went Out of Fashion

When an interviewer asks how you regularize a pre-training run, the answer that separates practitioners from textbook readers is: you almost certainly do not use dropout. This question surfaces regularly at mid-to-senior LLM engineer interviews whenever the conversation moves from architecture into training configuration. Getting it wrong signals that you are importing fine-tuning intuitions into a fundamentally different regime - and that you have not thought carefully about what pre-training actually is at a systems level. The stakes are higher than they appear: understanding why dropout is inappropriate leads directly to understanding what regularization tools do apply, and when.

Dropout randomly zeros a fraction $p$ of activations during each forward pass. During training, each activation unit is retained with probability $( 1 - p )$, and in inverted dropout (the standard modern implementation), activations are rescaled by $\frac{1}{1 - p}$ so that the expected magnitude is preserved at test time. The objective is to prevent co-adaptation: no single neuron can rely on the persistent presence of a specific upstream neuron, because that upstream unit may be zeroed on the next step. The network must learn redundant, distributed representations, and the ensemble of all exponentially many thinned sub-networks produces a model with lower generalization error.

This argument rests on one critical assumption: the model can overfit the training data. Given sufficient capacity and insufficient data, the model will memorize training examples and fail to generalize. Dropout introduces stochastic noise during training that prevents this memorization. In image classification with tens of thousands of examples, in question answering with carefully curated datasets, this assumption is almost always valid.

Pre-training for LLMs violates this assumption completely. A 7B-parameter model trained on 1T tokens sees approximately 143 tokens per parameter over the entire training run. Each individual training example - a web page, a book passage, a code file - is seen exactly once. There is no mechanism by which the model can memorize specific examples, because each example occupies a negligible fraction of the parameter update history. Overfitting, in the classical sense, does not occur.

Dropout is a regularizer that prevents overfitting; LLM pre-training is a regime where overfitting cannot occur; therefore dropout is solving a problem that does not exist.

The empirical evidence is unambiguous. When you measure the gap between training loss and validation loss with and without dropout across pre-training runs at multiple scales, the gap is negligible in both conditions. Neither run overfits. The difference between the two conditions is not in generalization but in optimization efficiency: the run with dropout shows a consistently higher training loss throughout, because each gradient step is computed on a stochastically degraded view of the network.

At high learning rates with large-batch training - the standard regime for pre-training frontier models - the gradient signal is already averaged over thousands of tokens per step. Adding dropout atop this introduces additional variance without providing any compensating benefit. The network takes longer to converge to any given loss level, and the final training loss is worse. The validation loss tracks the training loss, confirming that the degradation is not a regularization win but a pure optimization loss.

The contrast with fine-tuning is worth making explicit. When you fine-tune a pre-trained model on a task-specific corpus of tens of thousands of examples and run for multiple epochs, the model can absolutely overfit. The architecture is identical; the training regime is completely different. Dropout at $p = 0 . 05$-$0 . 1$ is often beneficial in that setting, particularly in the later transformer layers where task-specific representations are being adapted. Applying the same treatment to pre-training is a category error that conflates two regimes with fundamentally different statistical properties.

There is also a hardware efficiency argument. Dropout requires generating a random binary mask at every dropout site per forward pass. With 32 layers, two dropout sites per layer (post-attention and post-FFN), and a batch of $4 \times 10^{6}$ tokens per step, that is $2 . 56 \times 10^{8}$ random number generations per step. This is not the dominant cost, but it is nonzero, and across 200,000-500,000 training steps on thousands of accelerators, it accumulates into compute that could be spent elsewhere.

A Worked Example

Consider a 7B-parameter decoder-only model trained for one epoch on 1T tokens, with global batch size 4096 sequences of length 4096 tokens ($\approx 4 \times 10^{6}$ tokens per step), running for approximately 245,000 steps on H100s.

Configure two runs: Run A with dropout=0.0, Run B with dropout=0.1 applied at all post-attention and post-FFN sites across 32 layers.

At step 50,000 (roughly 20% through training), Run B has a loss $\approx 0 . 03$-$0 . 05$ nats higher than Run A. This deficit is not because Run A is overfitting - the validation losses track training losses identically in both runs. Run B’s gradient updates are simply less informative: each step sees a different stochastic degradation of the network, injecting variance that the optimizer must average out over subsequent steps.

At the end of training, the train-to-validation loss gap in both runs is $< 0 . 01$ nats - confirming that neither run is overfitting. But Run B’s final training loss is $0 . 02$-$0 . 05$ nats worse than Run A’s. That delta, small in absolute terms, translates into measurable downstream degradation on benchmarks such as MMLU and HellaSwag. You paid a dropout tax and received no regularization dividend.

What You Actually Decide in Practice

• Set dropout to zero for pre-training, explicitly. Do not rely on defaults. Many open-source training harnesses (Megatron-LM, NeMo, earlier versions of GPT-NeoX) have nonzero dropout defaults inherited from supervised learning use cases. Always override with dropout=0.0 in your pre-training config.

• Re-enable dropout for fine-tuning on small datasets. If your fine-tuning corpus has fewer than a few hundred thousand examples and you run more than one epoch, dropout at $p = 0 . 05$-$0 . 1$ applied to the upper layers is a reasonable starting point. The overfitting regime has returned.

• Distinguish dropout from other stochastic training techniques. Stochastic depth (dropping entire residual blocks with some probability) and attention dropout (zeroing attention weights rather than activations) are distinct from standard dropout. Both have niche use cases in pre-training research, but neither is standard practice for production runs.

• One-epoch assumption is load-bearing. If your pre-training setup uses data repetition - for example, a high-quality curated subset upsampled 10$\times$ relative to web data - those repeated samples do create an overfitting surface. In practice, this is handled via data-mixing ratio tuning rather than dropout, because dropout’s effect is uniform across all tokens regardless of whether they are repeated.

• Audit GPT-3 era codebases carefully. GPT-3 used residual dropout and embedding dropout during pre-training - a choice made before the community had accumulated the ablation evidence against it. Codebases derived from GPT-2 or GPT-3 may silently retain these settings.

How This Shows Up in Interviews

• “Walk me through how you’d configure regularization for a 7B pre-training run.” A strong answer distinguishes pre-training from fine-tuning immediately. For pre-training: dropout zero, weight decay enabled (see next subsection), z-loss for softmax stability, gradient clipping. For fine-tuning on small data: dropout re-enabled in upper layers, early stopping, potentially higher weight decay.

• “Why don’t modern LLMs use dropout during pre-training?” Answer skeleton: (1) pre-training is one pass over the data - the model cannot overfit individual examples; (2) the overfitting risk that motivates dropout does not exist; (3) dropout adds gradient noise without any compensating benefit; (4) empirically, the train-to-val gap is negligible with or without it, but the absolute training loss is worse with dropout.

• Senior follow-up: “GPT-3 used dropout during pre-training. Was that a mistake, or is there something you’re missing?” A senior answer avoids the trap of retroactively labeling it a mistake. At the time GPT-3 was trained (early 2020), the empirical case that pre-training overfitting was a non-issue at 175B parameters had not been established through public ablations. The team also used dropout partly as a training stability hedge, not purely for regularization. The lesson is that best practices in this field are empirically derived and lag model scale - what was reasonable caution in 2020 became unnecessary overhead by 2022. A senior candidate also notes that GPT-3’s dropout rates were small ($p = 0 . 1$) and their impact on the final model was likely minor compared to other training decisions.

Key Takeaways

• Dropout prevents overfitting; pre-training on internet-scale data for one epoch has no overfitting risk; therefore dropout has no benefit during pre-training.

• Removing dropout is now universal consensus among frontier labs - set it explicitly to zero and verify your codebase defaults.

• Dropout re-enters the picture for fine-tuning on small corpora with multiple epochs.

• The diagnostic: if adding dropout changes your train-to-val gap, overfitting is happening; if it does not change the gap but raises training loss, dropout is only hurting you.

• The broader principle: always verify that a regularization technique is solving a problem that actually exists in your training regime before applying it.

4.4.2 Weight Decay and Learning-Rate Interactions

If the previous subsection convinced you that pre-training does not need regularization because overfitting is not a risk, then weight decay is about to complicate your picture. Nearly every frontier LLM pre-training run uses weight decay - and weight decay is textbook L2 regularization. This apparent contradiction catches engineers off guard, and resolving it correctly requires understanding a mechanism that has nothing to do with the bias-variance tradeoff. When interviewers push past “we use weight decay to prevent overfitting” and ask why it actually helps when overfitting is not occurring, the answer reveals whether you have genuinely internalized the dynamics of large-scale optimizer behavior or are pattern-matching to supervised learning intuitions.

Weight decay adds a penalty to the loss proportional to the squared L2 norm of model parameters:

In standard SGD, this is equivalent to subtracting $\lambda \theta_{i}$ from each parameter at every step - hence the name weight decay. In AdamW, the correct implementation for adaptive optimizers, the decay is applied directly to the parameters after the Adam update rather than being folded into the gradient:

where $m_{t}$ and $v_{t}$ are the first and second moment estimates. This decoupling matters: plain Adam with an L2 gradient penalty scales the effective decay rate per-parameter by $1 / \sqrt{v_{t}}$, making it non-uniform and often counterproductive at large scale.

The standard motivation for weight decay in supervised learning is regularization: large weights signal over-reliance on specific features, and penalizing them pushes the model toward simpler hypotheses that generalize better. In LLM pre-training, this framing is wrong. When you sweep weight decay across typical values ($\lambda \in \{ 0 . 01 , 0 . 1 , 0 . 3 \}$) and hold all else equal, the train-to-validation loss gap does not meaningfully change. Overfitting is not occurring either way. What changes instead is the absolute training loss - specifically, how much further the optimizer can descend during the learning rate cooldown phase at the end of training.

In LLM pre-training, weight decay is not a regularizer; it is an optimizer-conditioning mechanism that synergizes with the learning rate schedule to achieve lower final training loss.

The mechanism becomes visible when you study training loss curves under different combinations of weight decay and learning rate schedule shape. Consider a cosine decay schedule that takes a peak learning rate $\eta_{\text{max}}$ down to a minimum $\eta_{\text{min}} \approx \eta_{\text{max}} / 10$ over the full training run.

Through the high-learning-rate phase - roughly the first 80% of training - weight decay acts as a headwind. The optimizer is making large parameter updates, and weight decay is simultaneously shrinking all parameters toward zero. The net effect is that the model with weight decay trains more slowly during this phase: at any given step count, its training loss is slightly higher than the zero-weight-decay baseline. This is exactly the behavior you would expect from any regularization that conflicts with the gradient signal.

But the picture reverses in the cooldown phase. As $\eta$ drops from $\eta_{\text{max}}$ toward $\eta_{\text{min}}$, the magnitude of gradient updates shrinks, and the optimizer begins to settle into a local basin. The model without weight decay has allowed parameters to grow large throughout training, pulled by whatever gradient directions were most informative at high learning rates. Large parameters create sharp, irregular loss landscape curvature: the Hessian eigenvalues in high-magnitude directions are large, making it hard for small gradient steps to traverse the terrain cleanly.

The model with weight decay has maintained smaller parameter magnitudes throughout. Weight decay has effectively been imposing a soft bound on the parameter scale, trading some optimization speed during the high-LR phase for a smoother, better-conditioned parameter neighborhood at the end. When the learning rate cools, this model sits in a flatter basin, and small gradient steps can navigate further down the loss landscape. The result is a lower final training loss - and, critically, a lower validation loss in lockstep, since overfitting is not occurring. The benefit is real and it is an optimization quality improvement, not a regularization artifact.

A Worked Example

You are training a 13B model with AdamW: $\beta_{1} = 0 . 9$, $\beta_{2} = 0 . 95$, $\epsilon = 10^{- 8}$, peak learning rate $\eta_{\text{max}} = 3 \times 10^{- 4}$, cosine decay to $\eta_{\text{min}} = 3 \times 10^{- 5}$ over 500,000 steps. Global batch size: 2048 sequences of 4096 tokens ($\approx 8 . 4 \times 10^{9}$ tokens per epoch).

Run A: $\lambda = 0 . 0$. Run B: $\lambda = 0 . 1$.

Steps 1-400,000 (high-LR phase): Run A loss $\approx 0 . 02$-$0 . 03$ nats lower than Run B at each checkpoint. Run B is paying the weight-decay headwind tax.

Steps 400,000-500,000 (cooldown phase, $\eta$ decaying from $\sim 10^{- 4}$ to $3 \times 10^{- 5}$): Run B begins to close the gap. The gradient steps are now small enough that the well-conditioned parameter landscape of Run B allows more efficient descent.

Final result: Run B ends with a training loss $\approx 0 . 02$-$0 . 05$ nats lower than Run A. The train-to-val gap in both runs: $< 0 . 01$ nats. The conclusion is unambiguous - weight decay improved optimizer convergence, not generalization.

The optimal $\lambda$ for a given run depends on $\eta_{\text{max}}$ and the schedule shape. At $\eta_{\text{max}} = 3 \times 10^{- 4}$, $\lambda = 0 . 1$ is a strong default. If you push $\eta_{\text{max}}$ to $10^{- 3}$ (aggressive but used in some runs), reduce $\lambda$ to $0 . 05$ to avoid making the high-LR phase too slow.

What You Actually Decide in Practice

• Use AdamW, not Adam with L2 penalty. The distinction is not cosmetic. With plain Adam and an L2 gradient term, the effective per-parameter decay is $\lambda / \sqrt{v_{i} + \epsilon}$. Parameters with high gradient variance get less decay; parameters with low gradient variance get more. This breaks the intended uniform conditioning effect. AdamW applies $\lambda$ uniformly and directly to the parameter values after the Adam step. Verify your framework is using the correct implementation - some older codebases still default to the wrong one.

• Exclude embeddings and normalization parameters from weight decay. Decaying embedding matrices shrinks all token vectors uniformly toward zero, collapsing the diversity of token representations independently of their actual usage frequency. RMSNorm scale parameters and any learned bias terms in normalization layers are scale hyperparameters, not weights in the usual sense, and should also be excluded. Modern frameworks support parameter groups with separate hyperparameters for this purpose.

• $\lambda = 0 . 1$ is the empirical consensus for most configurations. Values below $0 . 01$ produce negligible conditioning effects. Values above $0 . 3$ impose a headwind in the high-LR phase severe enough to outweigh the tail-of-training benefit at standard learning rates. The effective range is $0 . 05$-$0 . 2$; if you are uncertain, start at $0 . 1$.

• Weight decay interacts with schedule shape, not just magnitude. With a trapezoidal or WarmupStable-Decay (WSD) schedule - long stable phase at $\eta_{\text{max}}$ followed by a sharp decay - the benefit of weight decay is concentrated in the final decay phase. You can think of the stable phase as building up the conditioned parameter landscape, and the decay phase as harvesting it. With cosine decay, the interaction is more gradual but qualitatively the same.

• Gradient clipping is a separate intervention and must be configured independently. Gradient clipping ($\| \mathbf{g} \|_{2} \leq \tau$, typically $\tau = 1 . 0$) prevents individual catastrophic gradient steps and is standard alongside weight decay. The two solve different problems: clipping prevents explosive single steps; weight decay conditions the parameter space over the full training run. Never mistake one for the other in an interview.

How This Shows Up in Interviews

• “If pre-training doesn’t overfit, why does every frontier model use weight decay?” The correct answer immediately reframes: weight decay in pre-training is not preventing overfitting. It is improving final training loss by conditioning the parameter landscape for more effective optimization during the learning rate cooldown. The train-to-val gap does not change with or without weight decay; the absolute loss level does. The benefit mechanism is optimizer dynamics, not regularization.

• “What’s the difference between Adam and AdamW, and why does it matter?” Skeleton: In Adam with L2, the weight decay term enters the gradient, which then passes through the adaptive scaling $1 / \sqrt{v_{i}}$. The effective decay per parameter varies with gradient history, breaking the intended conditioning effect. AdamW applies the decay directly to the parameter after the Adam update, making it uniform across all parameters. At large model scale, this implementation difference produces measurably different training dynamics, not just theoretical differences.

• Senior follow-up: “Can you give an intuitive explanation for why weight decay and cosine LR decay interact the way they do - beyond just citing empirical results?” A senior answer builds the following chain: weight decay maintains a tighter distribution of parameter magnitudes throughout training, which constrains the loss landscape curvature around the current parameter values. At high learning rates, the optimizer is taking large steps and the curvature constraint matters less - gradient direction dominates. As the learning rate decreases, the optimizer’s step size falls into the regime where local curvature is the binding constraint. A model sitting in a smoother neighborhood (achieved through weight decay) can take those small steps more efficiently and descend further. Formally, this connects to the relationship between weight regularization and implicit sharpness reduction: weight decay is known to reduce the dominant eigenvalue of the Hessian in over-parameterized models, and low-sharpness minima are exactly the basins that small-step optimizers can most effectively exploit.

Key Takeaways

• Weight decay in LLM pre-training is an optimizer-conditioning mechanism, not a regularizer - it improves final training loss, not the train-to-val gap.

• Always use AdamW with explicit $\lambda$; never use plain Adam with an L2 gradient penalty for large-scale pre-training.

• Exclude token embeddings and normalization parameters from weight decay; apply it only to projection weights and attention matrices.

• Standard value is $\lambda = 0 . 1$ for most configurations; reduce to $0 . 05$ if using an aggressive peak learning rate above $10^{- 3}$.

• The benefit concentrates in the tail of training, during LR cooldown - the diagnostic is that a model trained with weight decay should finish lower in training loss than one without it, while both should show the same negligible train-to-val gap.