Chapter 13 Predictable Scaling for Interview Answers

13.1 The Three Canonical Scaling Laws

13.1.1 Data Scaling and Non-Parametric Rates

An interviewer at a frontier lab will often open the scaling-laws deep-dive with a deceptively simple question: “If I double my training data, how much does the loss drop?” The answer is not “a lot” or “it depends.” The answer is a precise power-law relationship, and if you can derive the exponent from first principles on a whiteboard, you immediately signal a level of understanding that separates senior candidates from the pack. This topic appears at the staff and principal level, but even L5 candidates are expected to know the empirical fact; the derivation is what earns you the senior signal.

Data scaling is the observation that, on a log-log plot, test loss decreases linearly as a function of dataset size. Formally, if we denote the dataset size in tokens as $D$ and the cross-entropy loss as $L$, then in the scaling regime we find:

where $D_{0}$ is a dataset-specific constant and $\alpha > 0$ is the scaling exponent. Taking logarithms of both sides:

This is a line with slope $- \alpha$ in log-log space. The empirical finding, established robustly by Hestness et al. (2017) and confirmed at scale by Kaplan et al., is that this linear relationship holds across many orders of magnitude of data, across different model architectures, and across domains including language, speech, and machine translation. The exponent $\alpha$ is typically in the range $[ 0 . 05 , 0 . 35 ]$ depending on the task-much smaller than what naive statistical theory would suggest. The model size $N$ is always chosen to be large enough that the data is not saturating the model; otherwise, you would observe premature flattening as you enter the irreducibleerror regime.

Log-log linearity between dataset size and test loss is the most robustly observed empirical fact in largescale language modeling, and its exponent encodes the effective intrinsic dimensionality of the learning task.

The power-law decay is not magic-it has a grounded theoretical explanation rooted in non-parametric statistics. Consider the simplest possible estimation task: estimating the mean $\mu$ of a Gaussian distribution from $n$ i.i.d. samples. The mean estimator $\hat{\mu} = \frac{1}{n} \sum x_{i}$ has squared error:

This is polynomial decay with exponent $- 1$ in $n$. But language models are not estimating a scalar mean-they are fitting an arbitrarily complex function over a very high-dimensional input space. For a smooth regression target $f$ on a $D$-dimensional domain, even the optimal non-parametric estimator (e.g., a histogram regressor) achieves squared error scaling as:

In log-log terms, the slope is $- 1 / D$. A task with effective intrinsic dimensionality $D = 10$ gives a slope of $- 0 . 1$; dimensionality $D = 3$ gives $- 0 . 33$. This matches the empirically observed range of scaling exponents almost exactly: Hestness reports $- 0 . 13$ for machine translation, $- 0 . 30$ for speech, and $- 0 . 095$ for language modeling. Language modeling has a shallower slope because natural language is an extraordinarily high-dimensional distribution-the effective dimensionality of “predict the next token given all of human text” is enormous. The prior alternative was to rely on generalization bounds (VC dimension, Rademacher complexity), which provide rates like $1 / \sqrt{n}$ for agnostic learning; these are upper bounds that are too loose to be useful predictions, and empirical scaling laws are the practitioner’s replacement for them.

A Worked Example

Suppose you are fitting a smooth regression function on a 2D input space with $n$ total samples placed uniformly at random. Partition the unit square into $k \times k$ cells, assigning one estimated value per cell. With $k^{2}$

total cells, each cell receives $n / k^{2}$ samples on average.

The estimation error inside each cell is $\sigma^{2} / ( n / k^{2} ) = \sigma^{2} k^{2} / n$ (variance of a local mean estimate). The approximation error-how well a $k \times k$ grid captures a smooth function-scales as $1 / k^{2}$ for a function with bounded second derivatives. Balancing these two:

Plugging back in, the total error scales as $k^{- 2} \sim n^{- 1 / 2} = n^{- 1 / D}$ with $D = 2$. For a $D$-dimensional space the argument generalizes to:

Error $\sim n^{- 2 / ( D + 4 )}$ (exact for Hölder smooth functions)

which for large $D$ simplifies to $n^{- 1 / D}$ in the dominant term. Now apply this intuition to language modeling with an effective dimensionality of roughly $D \approx 10$: we would predict a scaling exponent of $- 0 . 10$, which is almost precisely the $- 0 . 095$ observed empirically for language modeling in Hestness et al. The model size must be fixed large enough that it is not the bottleneck-a 7B-parameter model trained in bf16 on an H100 cluster can serve this role for datasets up to a few hundred billion tokens before saturation becomes a concern.

What You Actually Decide in Practice

• Model size must exceed data saturation. When running data-only scaling experiments, fix the model large enough that the scaling curve stays in the power-law region. A rule of thumb: for every doubling of data, you want roughly $10 \times$ more parameters than what would be “Chinchilla-optimal” for that data size, so the model cannot overfit to the finite dataset.

• Data composition shifts the offset, not the slope. Kaplan et al. show that changing training data mixture (domain proportions) changes the vertical intercept of the log-log line but leaves the slope $\alpha$ nearly unchanged. This means data selection experiments-which source to mix in, what weight-can be validly run on small models and small data; the relative rankings transfer to scale.

• Repeated data hurts after roughly 4 epochs. When you have exhausted unique tokens, repeating the corpus has rapidly diminishing returns. Empirically the effective sample size grows much more slowly than the raw token count past 4 epochs of repetition. If your corpus allows only 2 epochs before saturation, adding lower-quality unique data is almost always better than a third pass over high-quality data.

• Do not confuse perplexity scaling with downstream-task scaling. The power-law relationship is clean and reliable for cross-entropy loss. Benchmark accuracy can be non-monotone, can show emergent jumps, and can rank architectures differently than perplexity does. Use perplexity scaling to make engineering decisions; validate on downstream tasks before committing resources at scale.

• The irreducible floor exists. Every task has an entropy floor below which no amount of data will push the loss-the Bayes error of the task. In practice, you will rarely be close to it for language, but in low-entropy domains (e.g., heavily templated text), you can hit it surprisingly early.

How This Shows Up in Interviews

• “I double the number of training tokens. How does the validation loss change?” A strong answer names the power-law: loss drops by a factor of $2^{\alpha}$ where $\alpha \approx 0 . 05$-$0 . 30$ depending on the task. For language, $\alpha \approx 0 . 1$, so doubling tokens gives a $\approx 7 \%$ relative improvement in loss. The candidate should also mention that the model must be large enough to remain in the scaling regime.

• “Why is the empirical scaling exponent so much smaller than $1 / \sqrt{n}$?” The strong answer explains non-parametric rates: $- 1 / \sqrt{n}$ is the rate for estimating a scalar or fitting a one-dimensional function (slope $- 0 . 5$). For a $D$-dimensional task the exponent is $- 1 / D$; language modeling has very high effective dimensionality, so the slope is shallow. The log-log linearity itself comes from the polynomial structure of non-parametric error bounds.

• Senior follow-up: “If you changed the training data mixture-say, you doubled the fraction of code-how would that affect the scaling curve?” The senior answer: the slope $\alpha$ is largely unchanged because it reflects the intrinsic difficulty of the learning task. The offset shifts-more code would lower the floor for code benchmarks while potentially raising it for other domains. You can measure this offset shift cheaply by training small models ($\sim$1B parameters, $\sim$10B tokens) and fitting a line; the offset difference at small scale extrapolates reliably to large scale, which is the entire point of data-composition scaling experiments.

Key Takeaways

• On a log-log plot, test loss versus dataset size is linear in the scaling regime; the slope is the scaling exponent $\alpha$.

• The exponent $\alpha$ for language modeling is $\approx 0 . 1$, reflecting high effective intrinsic dimensionality; it arises naturally from non-parametric rates of the form $n^{- 1 / D}$.

• The model must be large enough that data is not saturating it-otherwise you leave the power-law regime early.

• Data composition changes the offset of the scaling curve but not its slope, so data-selection experiments are cheap to run at small scale.

• Repeated data beyond $\sim$4 epochs yields rapidly diminishing returns; prefer new lower-quality unique tokens over additional passes over a high-quality corpus.

13.1.2 Model and Compute Scaling

“If I give you a fixed compute budget, how would you split it between model size and training data?” is the single most commonly asked quantitative question in frontier-lab LLM design interviews. The answer comes from the Chinchilla analysis, and getting it right-including why Kaplan’s earlier estimate was off and what the three estimation methods actually measure-is what separates a candidate who memorized a ratio from one who understands the underlying machinery. This question appears at every seniority level; the senior version asks you to derive the optimal trade-off from scratch rather than recite it.

Model and compute scaling extends data scaling to a two-variable problem. The loss $L$ depends jointly on model size $N$ (non-embedding parameters) and dataset size $D$ (tokens):

Here $A$, $B$, $\alpha$, $\beta$ are empirically fitted constants and $L_{\infty}$ is the irreducible entropy of the data distribution. This functional form is additive: the model-size term and the data-size term contribute independently to reducible loss. The key insight is that for a fixed compute budget $C$ (in FLOPs), training cost scales approximately as $C \approx 6 N D$ (the factor of 6 counts both forward and backward passes per token per parameter). Given that constraint, you want to minimize $L ( N , D )$ subject to $N D = C / 6$.

Taking the gradient of the Lagrangian and setting it to zero yields the compute-optimal or Chinchillaoptimal scaling:

where the Chinchilla estimates give $a \approx b \approx 0 . 5$, implying that $N$ and $D$ should be scaled in equal proportion as compute grows. The resulting rule of thumb is roughly 20 tokens per parameter: a 70B-parameter model is compute-optimal when trained on $\sim$1.4 trillion tokens.

Compute-optimal training balances model size and token count in a roughly 1:20 parameter-to-token ratio; skewing either way wastes FLOPs.

Kaplan et al. (2020) were the first to analyze joint model-data scaling, but they significantly underestimated the optimal token-to-parameter ratio-their estimate was closer to 5-10 tokens per parameter. The primary culprit was cosine learning rate schedules. A cosine schedule cannot be truncated mid-run; if you stop training early (before the full cosine cooldown), the model’s effective loss is higher than it would be at the same number of steps with a properly completed schedule. Kaplan’s experiments used runs truncated at various compute levels for comparison, which systematically overestimated the performance of large models trained on little data and underestimated small models trained longer. Chinchilla used isoFLOP profiles-multiple models trained to the same total compute budget, each completing its full learning rate schedule-which avoids this artifact.

The additive functional form of $L ( N , D )$ also encodes a crucial architectural observation: not all parameters count equally. Embedding parameters (the token embedding matrix and positional embeddings) do not follow the same power-law behavior as non-embedding parameters. When embedding parameters are included in $N$, the scaling curve bends and loses its clean log-log linearity. Kaplan reports this explicitly: analysis should be conducted on non-embedding parameter counts. Similarly, in sparse models like mixture-of-experts, the number of activated parameters per token, not the total parameter count, governs the effective $N$ in the scaling law.

A Worked Example

Suppose your compute budget is $C = 6 \times 10^{23}$ FLOPs-roughly the budget for a mid-tier production training run on a few hundred H100s over several weeks. Using $C \approx 6 N D$:

Chinchilla’s compute-optimal condition says $N^{\ast} \approx \Bigl( \frac{C}{6} \cdot \frac{\alpha}{\beta} \Bigr)$. With $\alpha \approx \beta$ (Chinchilla methods 1 and 2 both give roughly equal exponents), this simplifies to:

and

That gives roughly 1 token per parameter-but this is for the abstract mathematical optimum ignoring the 20:1 convention. The discrepancy is because the 20:1 rule comes from the Chinchilla paper’s actual constant fits ($A$, $B$, $\alpha$, $\beta$), not from the $\alpha = \beta$ simplification alone. With Chinchilla’s fitted constants, the compute-optimal line for a 7B-parameter model falls at approximately 140B tokens, and for a 70B model at approximately 1.4T tokens-the 20:1 ratio.

The three methods Chinchilla uses to estimate this ratio are: (1) minimum-envelope over training curves across compute levels, which finds the lower envelope of loss vs. compute for models of varying size; (2) isoFLOP profiles, which sweep model size at each fixed compute level and find the minimum of a fitted quadratic; and (3) parametric fit to the full $L ( N , D )$ surface via least-squares. Methods 1 and 2 agree closely (0.50, 0.50 exponents); method 3 originally gave anomalous results because of an off-center residual in the curve fitting, a flaw later identified by Epoch AI’s forensic replication of the plots.

What You Actually Decide in Practice

• Training-optimal is not inference-optimal. Chinchilla optimizes training FLOPs. Production systems care about inference cost-a smaller model with the same quality at deploy time is far cheaper to run at scale. This is why modern models have drifted far from 20 tokens per parameter: Llama 3 (8B) was trained on 15 trillion tokens ($\sim$1875 tokens per parameter), and Qwen models have reached 30 trillion tokens. When your product serves millions of requests per day, the right ratio is determined by inference cost, not training cost.

• Exclude embedding parameters from $N$. Always use non-embedding parameter counts when fitting scaling laws or comparing model sizes. Embedding tables can be very large (e.g., a 128k vocabulary at 4096 dimensions adds $\sim$500M parameters) and do not improve loss at the same rate as attention and feedforward parameters.

• For MoE models, use activated-parameter count. In sparse mixture-of-experts architectures, only a fraction of parameters are active per token. The effective $N$ for scaling-law purposes is the number of activated parameters per token, not the total parameter count. Several papers have proposed “dense equivalents” to normalize MoE model sizes for comparison.

• Learning rate schedules must complete. When running scaling-law pilot experiments with cosine schedules, train each run to completion (full cooldown) before comparing. Truncated runs systematically misrank large-model versus long-training options. Constant or linear decay schedules sidestep this problem and are increasingly preferred for scaling experiments.

• Downstream task scaling is noisier than perplexity scaling. The compute-optimal analysis applies cleanly to cross-entropy loss. If you are trying to optimize for a specific benchmark, scaling predictions become unreliable-emergence effects and saturation are common. Use perplexity scaling to allocate resources and downstream benchmarks to validate the final model.

How This Shows Up in Interviews

• “If you had a fixed compute budget, how would you split it between model size and training tokens?” The strong answer names Chinchilla: roughly 20 tokens per parameter is training-optimal. Then the candidate immediately qualifies it: if the model will be deployed at high request volume, you should overtrain a smaller model-the inference savings over the model’s lifetime dominate the extra training cost. This distinction is what separates rote recall from engineering judgment.

• “Why did Kaplan underestimate the optimal token count compared to Chinchilla?” Strong answer: cosine learning rate schedules require completing the full cooldown to yield a valid checkpoint. Kaplan’s multi-model comparison used truncated runs, which made large models trained on little data look artificially better. Chinchilla’s isoFLOP method trained all runs to completion before comparing, removing the artifact.

• Senior follow-up: “Walk me through how you would run an isoFLOP experiment to estimate the compute-optimal model size for your team’s next training run.” The senior answer sketches: (1) pick 5-7 compute budgets spanning 2-3 orders of magnitude below your target; (2) at each budget, train 5-8 model sizes ranging from $\sim$0.3$\times$ to $\sim$3$\times$ the Chinchilla-optimal size for that budget; (3) fit a quadratic to the loss-vs.-model-size curve at each fixed compute; (4) extract the minimum and plot optimal $N$ vs. $C$ on a log-log axis; (5) fit a power law and extrapolate to your target compute. Emphasize that the entire pilot study uses $<$1% of the target compute budget and should complete in days, not weeks.

Key Takeaways

• The Chinchilla functional form $L ( N , D ) = A / N^{\alpha} + B / D^{\beta} + L_{\infty}$ describes joint model-data scaling; minimizing it subject to a fixed compute budget $C \approx 6 N D$ gives the training-optimal allocation.

• Chinchilla’s result is approximately 20 tokens per parameter for training-optimal runs; this is far below what modern production models use because inference cost dominates at deployment scale.

• Kaplan’s earlier (lower) estimate of the optimal token count was biased by truncated cosine learning rate schedules; isoFLOP experiments that complete the full cooldown correct this.

• Exclude embedding parameters from $N$ in all scaling analyses; they do not follow the same power law as attention and feedforward parameters.

• The isoFLOP method-sweeping model size at each fixed compute budget and extracting the minimum-is the most reliable and conceptually cleanest approach to estimating compute-optimal ratios for your specific task and architecture.

13.2 Using Scaling Laws to Make Design Decisions

13.2.1 Architecture Comparisons at Small Scale

Every year brings a new architecture claiming to beat the transformer. The interview question that follows is always the same: “How would you decide whether to adopt this for your next production training run?” The naive answer-train both at full scale and compare-is economically absurd. The right answer uses scaling laws to make that comparison at a fraction of the cost. This question appears at the senior level and above, and the candidate who can describe the procedure cold, with the right caveats, signals genuine production experience.

The scaling-law comparison procedure works as follows. You pick two or more candidate architectures and train instances of each across a range of compute budgets spanning two to three orders of magnitude-say, from $10^{18}$ to $10^{21}$ FLOPs. For each architecture, you measure the best achievable validation loss at each compute level (holding the other architecture fixed and sweeping over model sizes to get the optimal point). You then plot both curves on a log-log axis: log compute on the $x$-axis, log loss on the $y$-axis.

The crucial diagnostic is the relationship between the two curves:

• Parallel curves with a constant vertical gap: the better architecture is compute-efficient by a fixed multiplicative factor at every scale. If architecture A sits $\Delta$ log-loss units below architecture B across all compute levels, that translates to a constant factor speedup-architecture A achieves the same loss as B with a $k$-fold smaller compute budget, where $k = 10^{\Delta / \alpha}$ and $\alpha$ is the common scaling slope.

• Crossing curves: the ranking reverses at some compute level. The cheaper architecture may look better at small scale but lose at large scale, or vice versa. Crossing curves require extrapolation before you can make a decision.

A constant-factor offset in log-log scaling curves is the signature that an architecture advantage is real and will survive to your target compute budget.

The reason this procedure is reliable is that the slope of a scaling curve-the rate at which loss decreases per unit of log-compute-is largely determined by the task and data, not the architecture. What the architecture controls is the efficiency constant: how many FLOPs are needed to reach a given loss. A better architecture shifts the whole curve down (lower loss at every compute level) or equivalently left (same loss at lower compute). Because the slope is preserved, a gap measured at $10^{19}$ FLOPs extrapolates cleanly to $10^{23}$ FLOPs.

Contrast this with the alternative of just training at small scale and comparing accuracy numbers. A model with 10M parameters and 50M tokens trained on a narrow benchmark tells you almost nothing about relative quality at 70B parameters and 2T tokens. The scaling-law approach is cheap precisely because the parallelism of curves means small-scale information is informative about large-scale rankings.

The transformer-vs-LSTM comparison is the canonical example of this working well. LSTMs consistently sit $\sim$1 log-perplexity unit above transformers at every compute level, on a log scale. That gap corresponds to roughly a 10-15$\times$ compute penalty: to match a transformer trained on a given budget, an LSTM needs 10-15 times more FLOPs. Knowing this at the $10^{18}$ FLOP level means you never need to run the $10^{23}$ FLOP LSTM experiment to know the answer. Similarly, gated linear units and mixture-of-experts architectures showed reliable parallel downward shifts relative to the standard transformer baseline, which is precisely why they became the dominant modifications at frontier labs.

A Worked Example

Suppose you want to evaluate a new state-space model architecture against a standard transformer. You train both at five compute budgets: $C \in \{ 10^{18} , 3 \times 10^{18} , 10^{19} , 3 \times 10^{19} , 10^{20} \}$ FLOPs. At each $C$, you sweep over five model sizes (varying width and depth) and select the lowest validation loss achieved.

After plotting on a log-log axis, you observe:

• Both curves have approximately the same slope, $\alpha \approx - 0 . 08$ (consistent with language modeling data scaling).

• The transformer curve sits $\Delta \approx 0 . 05$ log-loss units below the SSM curve consistently across all five budgets.

• The curves are parallel-no crossing.

Interpreting the gap: a constant offset of $\Delta = 0 . 05$ in log-loss means the SSM needs a compute multiplier of $k = 10^{\Delta / \alpha} = 10^{0 . 05 / 0 . 08} \approx 10^{0 . 625} \approx 4 . 2$ times more FLOPs to match the transformer at any target loss. For a $5M training run, the SSM would require an equivalent $21M run to achieve the same quality-a clear decision signal obtained for roughly 0.01% of the target budget.

Total pilot cost: five compute levels, two architectures, five model sizes each = 50 training runs. If your target run is $C_{\text{target}} = 10^{23}$ FLOPs, your pilot spans $10^{18}$-$10^{20}$, which is at most $10^{20} / 10^{23} = 0 . 1 \%$ of the target. In practice, the pilot consumes even less because the small runs dominate compute.

What You Actually Decide in Practice

• Require at least two to three orders of magnitude of compute coverage in your pilot. A single data point per architecture is not a scaling law; you need enough range to confirm linearity and detect potential crossings. In my experience, five compute levels spanning three orders of magnitude is the minimum to feel confident.

• Crossing curves are a stop sign, not an answer. If the curves cross within your pilot range, you need to extrapolate carefully-fit power laws to each segment, determine where they diverge, and assess whether your target compute falls before or after the crossing. Extrapolation beyond the pilot range is unreliable unless the curves are clearly parallel by the end.

• Always compare on validation perplexity, not downstream benchmarks. Perplexity scales cleanly and predictably. Benchmark accuracy is noisy, nonlinear, and subject to emergence effects that can make a better architecture look worse at small scale. Use perplexity to select the architecture; validate downstream performance only after committing to the winner.

• The two modifications that reliably beat the standard transformer at every scale are gated linear units and mixture of experts. All other alternatives (linear attention, state-space models, recurrent variants) show either no reliable gap or task-specific wins that do not generalize. This is exactly what largescale architecture sweep papers found, and it directly explains the architectural choices in current frontier models.

• Exclude embedding parameters from all comparisons. Embedding matrices do not follow the same power-law scaling as attention and feedforward parameters. If you include them, the scaling curves bend and you lose the clean log-log linearity that makes the comparison reliable. Always report and compare on non-embedding parameter counts.

How This Shows Up in Interviews

• “A researcher proposes a new attention variant. How do you decide whether it is worth scaling up?” A strong answer describes the scaling-law comparison: train both architectures at 3-5 compute levels spanning 2-3 orders of magnitude, check whether the new variant shows a consistent parallel offset below the baseline. If yes, quantify the compute-efficiency gain. If curves cross, extrapolate and determine whether the advantage emerges before or after your target compute.

• “Why not just train the new architecture at a small scale and compare accuracy on a benchmark?” The strong answer explains why benchmark accuracy is an unreliable signal at small scale: architectures can rank differently on perplexity versus benchmarks, and emergent capabilities can appear discontinuously. Perplexity scaling is the reliable channel; benchmark validation comes only after the architecture decision is made.

• Senior follow-up: “What would you do if the scaling curves for two architectures crossed within your pilot range?” The senior answer goes beyond “extrapolate.” It explains that crossing curves mean one architecture is better in the data-limited regime and the other in the compute-rich regime; you need to characterize where your target compute falls relative to the crossing point. If the crossing happens before your target, the extrapolation is more reliable; if after, you may need to push your pilot to larger scale before committing. The candidate should also ask why the curves cross-is it a depth-width interaction? A saturation artifact? Understanding the mechanism determines whether the crossing is robust or an artifact of the comparison setup.

Key Takeaways

• Train each candidate architecture at 3-5 compute levels spanning 2-3 orders of magnitude; compare on validation perplexity, not benchmark accuracy.

• Parallel curves with a constant vertical gap confirm that the better architecture’s advantage will survive to your target compute; the gap size gives you the compute-efficiency multiplier.

• Crossing curves require extrapolation and deeper investigation before committing.

• The two architecture modifications with the strongest and most consistent evidence of compute-efficiency improvement are gated linear units and mixture of experts.

• Always use non-embedding parameter counts; embedding parameters break log-log linearity and skew comparisons.

13.2.2 Optimizer and Hyperparameter Selection

Choosing the wrong learning rate for a 70B model is not a minor inconvenience-it can cost weeks of H100 time and produce a model that is meaningfully worse than it should be. The right way to pick learning rates, batch sizes, and depth-to-width ratios is not to tune them at large scale; it is to verify at small scale that these hyperparameters either transfer directly or follow a predictable scaling law of their own. Understanding when each case applies is a senior-level topic that shows up in both design and deep-dive interviews.

The scaling-law mindset applied to hyperparameters rests on one observation: if the optimal value of a hyperparameter produces scaling curves that are parallel and non-crossing across model sizes, then the optimal value at small scale is the optimal value at large scale. You train a grid of small models varying the hyperparameter of interest, identify the optimal value at several compute levels, and check whether that optimum is stable.

Three hyperparameters deserve special treatment:

Aspect ratio (depth-to-width). Kaplan et al. show that for models between 50M and 1.5B parameters, the loss difference between aspect ratios from $\sim$10 to $\sim$100 is small-well under 5% relative. The basin is wide and flat. A single layer is genuinely bad; all reasonable aspect ratios above that are approximately equivalent. The practical upshot: pick anything in the range $d_{\text{model}} / n_{\text{layers}} \in [ 64 , 256 ]$ and you are in the safe zone at all scales.

Batch size and critical batch size. Doubling the batch size is equivalent to taking two gradient steps-but only below the critical batch size $B^{\ast}$. Above $B^{\ast}$, additional samples in the batch reduce useful gradient noise at a diminishing rate, and you are effectively paying full parallelism cost for less than full optimization benefit. The critical batch size itself scales with the loss target: lower loss (later in training) implies larger $B^{\ast}$, because the gradient landscape becomes smoother and you need more samples to reduce noise to a useful level. Empirically, $B^{\ast}$ follows a power law:

for some task-dependent $\gamma > 0$, where $\mathcal{L}$ is the current validation loss. This is why production training runs (Llama 3 is a public example) ramp batch size during training: early in training, loss is high and $B^{\ast}$ is small; later, loss is lower and $B^{\ast}$ is larger, so you can increase parallelism without sacrificing optimization efficiency.

Learning rate. Under the standard parametrization, the optimal learning rate decays with model width $d$ roughly as $\eta^{\ast} \propto 1 / d$. This means every time you double the width, you need to re-tune the learning rate. The alternative, maximal update parametrization ($\mu$P), reparametrizes the model so that the optimal learning rate is exactly stable across width. You tune it once at a small model (say, 50M parameters) and apply the same value to a 70B model.

Hyperparameter tuning and architecture selection should be done at small scale and extrapolated; the only reliable signal is whether the optimal value is stable across compute levels on a log-log plot.

The critical insight about hyperparameter stability is that it is not universal. Some hyperparameters-aspect ratio, feedforward ratio, attention head dimension-have wide, flat loss basins that barely move as scale increases. Others-learning rate under standard parametrization, batch size-require active management as you scale. Knowing which is which saves enormous amounts of compute.

The optimizer comparison follows the same logic as architecture comparison. Adam versus SGD produces a roughly constant-factor compute efficiency gap across scales, with Adam consistently requiring roughly $10 \times$ fewer FLOPs to reach the same loss. Since the gap is parallel (constant offset on log-log axes), you don’t need to run SGD at full scale to know the answer-a small-scale pilot is sufficient.

The $\mu$P approach deserves particular attention because it eliminates one of the most expensive unknowns in scaling: the learning-rate transfer problem. Standard parametrization initializes attention projection weights with $\mathcal{N} ( 0 , 1 / d )$ variance and applies the same learning rate to all layers regardless of width. As width grows, the gradient signal through the attention layers blows up without the $1 / d$ correction, forcing the learning rate down. $\mu$P corrects this by scaling initialization variance, per-layer learning rates, and the forward-pass output multiplier by appropriate powers of $d$. The result is a model where every layer’s activation magnitude is invariant to width, and consequently the optimal global learning rate does not change as you scale.

A Worked Example

Consider selecting the optimal learning rate for a standard transformer with $d_{\text{model}} \in \{ 256 , 512 , 1024 , 2048 , 4096 \}$ (corresponding roughly to 15M, 55M, 220M, 900M, and 3.5B non-embedding parameters). For each width, you run a grid of seven learning rates spaced in log scale from $10^{- 4}$ to $10^{- 1}$, train for a fixed number of tokens (say 10B), and record the minimum validation loss.

The optimal learning rate at each width:

Fitting a power law: $\eta^{\ast} \propto d^{- 1 . 0}$, exactly the $1 / d$ rule. Extrapolating to $d = 8192$ (a 14B-parameter model), you predict $\eta^{\ast} \approx 10^{- 4}$. This is reliable enough to use as a starting point, but still requires a validation run because the learning-rate surface is narrow enough that being off by $3 \times$ costs you real loss.

Under $\mu$P, the same pilot at $d = 256$ with $\eta^{\ast} = 3 \times 10^{- 3}$ gives you the value to use at $d = 8192$-no extrapolation required. The computational cost of the pilot is the same; the benefit is that you eliminate the large-scale validation run.

What You Actually Decide in Practice

• Treat aspect ratio as a low-stakes decision. Pick a depth-to-width configuration that keeps you in the range of 10-64 layers and a $d_{\text{model}}$ that gives reasonable hardware utilization (multiples of 128 for tensorcore alignment). The loss basin is wide enough that you will not see a meaningful degradation from non-optimal aspect ratios-spend your tuning budget elsewhere.

• Set your starting batch size at or below the critical batch size, then ramp. A practical heuristic: at the beginning of training (loss $\approx 3$-4 nats for language modeling), start with a batch size of 0.5-2M tokens. As loss drops to $\approx$2 nats (mid-training), double the batch size. As loss approaches your target (e.g., 1.5 nats), double again. This mirrors what Llama 3 and similar runs do and keeps you in the efficient regime throughout.

• Use Adam (or AdamW) by default; only switch if you have a scaling-law pilot showing a competing optimizer wins. The empirical gap between Adam and SGD is approximately a $10 \times$ compute factor in favor of Adam, consistent across task types and model sizes. There is no strong evidence that alternatives (Adafactor, Lion, etc.) reliably beat Adam across all scales; run a small pilot before committing.

• Adopt $\mu$P if your team has the engineering bandwidth. The payoff is largest when you expect to train many models at different scales-hyperparameter transfer eliminates re-tuning costs that compound across experiments. The cost is implementation complexity: $\mu$P requires modifying initialization variance, perlayer learning rate multipliers, and output scaling, all as functions of width. Mistakes are subtle. If your team is running a one-off training run, standard parametrization with a careful scaling law fit for learning rate is lower risk.

• Verify parallelism before extrapolating any hyperparameter. The scaling-law approach is only valid if the loss curves for different hyperparameter values are parallel. Run your pilot, plot loss-vs-compute for each hyperparameter value, and confirm visually that the curves do not cross. If they cross, you cannot safely extrapolate the small-scale optimum to large scale.

How This Shows Up in Interviews

• “How do you pick the learning rate for a new 70B model?” A strong answer gives two paths: (1) standard parametrization-fit a power law on $\eta^{\ast}$ vs. $d$ from a small-scale pilot, extrapolate to the target width using $\eta^{\ast} \propto 1 / d$, then validate with a short 1B-token warmup run; (2) $\mu$P-tune $\eta^{\ast}$ once on a 50M model, apply directly at 70B, skip the large-scale validation run. The candidate should note the engineering tradeoff: path (1) is simpler but requires re-tuning at each scale; path (2) eliminates re-tuning but requires careful implementation.

• “What is the critical batch size and how do you use it in practice?” The strong answer defines it as the batch size above which marginal gradient steps become less efficient than ideal (gradient noise no longer decreasing proportionally with batch size), explains that it scales inversely with loss (lower loss $\rightarrow$ larger critical batch size), and describes the ramp strategy: start small and increase batch size as training progresses, matching the growing $B^{\ast}$.

• Senior follow-up: “Explain $\mu$P and why it solves the learning-rate transfer problem.” The senior answer explains the mechanics: standard parametrization initializes weights with variance $\propto 1 / d$ in attention layers but applies a width-independent learning rate, causing gradient signal magnitude to scale with $d$ and forcing $\eta^{\ast} \propto 1 / d$ to compensate. $\mu$P explicitly scales initialization variance and per-layer update rates by powers of $d$ so that forward-pass activations and weight updates remain $O ( 1 )$ regardless of width. The consequence: the gradient update magnitude is invariant to width, so the global learning rate that minimizes loss at width 256 is numerically the same that minimizes loss at width 8192. The limitation: $\mu$P guarantees this in the infinite-width limit; in practice, there are finite-width corrections, and labs still run short validation runs to confirm transfer.

Key Takeaways

• Hyperparameter stability (optimal value constant across scale) or predictable scaling (optimal value follows its own power law) can both be established cheaply with a small-scale pilot; never tune at large scale without this prior.

• Aspect ratio has a wide, flat loss basin-anything between 10 and 100 layers at a reasonable width is approximately optimal at every model size.

• Batch size should stay at or below the critical batch size $B^{\ast}$, which grows as training loss decreases; ramp batch size progressively during training to stay efficient.

• Under standard parametrization, optimal learning rate scales as $\sim 1 / d$; $\mu$P eliminates this dependence, allowing exact learning-rate transfer from a tiny pilot model to the full-scale run.

• Adam outperforms SGD by roughly $10 \times$ in compute efficiency across scales; this gap is consistent enough that it does not need to be re-validated for each new task.