Appendix A Napkin Math for LLM Interviews

A.1 Parameter Counts and Memory Footprints

These formulas size the weight tensors and the memory budget before writing a single line of code. The parameter derivations and their implications for architecture decisions are developed in subsection 3.1.1; the precision trade-offs behind the training-memory figure are covered in subsection 9.3.1.

Formulas

Parameters per transformer layer. With hidden dimension $d$ , SwiGLU expansion $d_{f f} = \lfloor 8 d / 3 \rfloor$ , $N$ query heads, and $N_{\mathrm{kv}}$ key-value heads (MHA: $N_{\mathrm{kv}} = N$ ; GQA: $N_{\mathrm{kv}} < N$ ):

P_{\text{attn}} = d^{2} \biggl( 2 + \frac{2 N_{\mathrm{kv}}}{N} \biggr) \approx 4 d^{2} ( \text{MHA} , N_{\mathrm{kv}} = N ) \qquad \text{(A.1)}

P_{\text{mlp}} = 3 d d_{f f} ( \text{SwiGLU: three matrices} ) \qquad \text{(A.2)}

P_{\text{layer}} = P_{\text{attn}} + P_{\text{mlp}} \qquad \text{(A.3)}

Total parameters (embedding table included):

P = L \cdot P_{\text{layer}} + V \cdot d

Memory footprint:

M_{\text{inference} , \text{bf16}} = 2 P \text{bytes} \qquad \text{(A.4)}

M_{\text{training} , \text{AdamW}} \approx 16 P \text{bytes} \qquad \text{(A.5)}

The 16 B/param training figure: bf16 working copy (2 B) + fp32 master weights (4 B) + fp32 gradients (4 B) + AdamW first and second moments (4+4 B). Activation memory (batch $\times$ sequence length dependent) is excluded.

Worked Example - 7B Model

Config: d = 4096, N = 32 heads, d_h = 128, L = 32 layers,
        d_ff = 11008 = floor(8 * 4096 / 3), V = 32000, bf16

Attention per layer : 4 * 4096^2        =  67.1 M
MLP per layer       : 3 * 4096 * 11008  = 135.3 M
Layer total         :                     202.4 M

Body (32 layers)    : 32 * 202.4 M      =   6.48 B
Embedding table     : 32000 * 4096      =   0.13 B
Grand total         :                   ~   6.6 B  (LLaMA-1 7B = 6.74 B)

Inference bf16      : 6.7e9 * 2         =  13.4 GB
Training AdamW      : 6.7e9 * 16        = 107.2 GB
Largest model on 8 x H100 (no activations):
  8 * 80 GB / 16 B/param = 40 B params (rough ceiling)

Rules of Thumb

$\text{MLP} \approx$ 2 $\times$ attention per layer for SwiGLU models (135 M vs. 67 M at $d = 4096$ ); the MLP holds $\sim$ 67% of each layer’s parameters.
Inference (bf16): $\approx$ 2 GB per billion parameters. A 70 B model needs $\approx$ 140 GB.
Training (AdamW): $\approx$ 16 GB per billion parameters. A 7 B model needs $\approx$ 107 GB before activations.
Ceiling on $n$ H100s (80 GB each): $n \times 80 / 16 = 5 n$ billion parameters (ignoring activations).
GQA with $N_{\mathrm{kv}}$ KV heads: reduces KV cache and KV weight contribution by factor $N / N_{\mathrm{kv}}$ . Use $N_{\mathrm{kv}} = N / 8$ (group size 8) as the production default for models $\geq$ 7 B.

Caveats

Bias terms are omitted; all modern frontier models drop them in attention and MLP.
For vanilla ReLU/GeLU MLPs use $d_{f f} = 4 d$ , not $\lfloor 8 d / 3 \rfloor$ . Mixing the two conventions inflates the count by $\sim$ 50%.
Activation memory scales with batch size $\times$ sequence length and can rival weight memory at large batch sizes; use gradient checkpointing to trade it back for compute.
Embedding weights are sometimes tied between input and output projections, halving the $V \cdot d$ contribution.

A.2 FLOPs per Forward and Backward Pass

FLOPs are the currency of training cost estimates and the input to every napkin-math scheduling question. The arithmetic intensity framework that connects FLOPs to hardware utilization is in subsection 8.2.1; isoFLOP analysis and Chinchilla compute-optimal ratios are in chapter 14.

Formulas

Single matrix multiply $A_{m \times k} \cdot B_{k \times n}$ :

\text{FLOPs} = 2 m k n

All other operations (softmax, layer norm, element-wise activations) are linear in tensor size and negligible compared to the matrix multiplications for large $d$ .

Forward pass (attention + MLP; dominant terms for $T \ll d$ ):

C_{\text{fwd}} \approx 2 N_{T} P

where $N_{T} = B \times T$ is the total number of tokens in the batch and $P$ is the parameter count. The attention quadratic adds $4 B L T^{2} N_{\mathrm{kv}} H$ FLOPs; this becomes comparable to the linear terms when $T \gtrsim d$ (e.g., $T = 8192$ , $d = 4096$ ).

Backward pass (parameter gradients $+$ activation gradients):

C_{\text{bwd}} \approx 4 N_{T} P ( \approx 2 \times C_{\text{fwd}} )

Full training step:

C_{\text{step}} \approx 6 N_{T} P

Model FLOPs Utilization (MFU):

C / t

MFU $=^{\text{step} \text{step}}$ peak FLOP/s (dense, no sparsity)

Worked Example - 70B Model, 15 T Tokens, 1024 H100s

Total training FLOPs = 6 * 70e9 * 15e12  = 6.3e24

H100 bf16 peak (dense, no sparsity)       = 989 TFLOP/s
At MFU = 0.5  : 989e12 * 0.5             = 4.945e14 FLOP/s per GPU
1024 GPUs     : 1024 * 4.945e14          = 5.06e17  FLOP/s total

Training time = 6.3e24 / 5.06e17 = 1.245e7 s  ~  144 days

Sanity check (8 H100s, 1 week, MFU = 0.5):
  FLOPs = 8 * 4.945e14 * 7 * 86400       = 2.4e21
  Chinchilla-optimal params (N_T = 20P):  2.4e21 / (6 * 20) ~ 20B params

Rules of Thumb

One training step: $C_{\text{step}} \approx 6 N_{T} P$ FLOPs.
H100 at MFU 0.5 (bf16 dense): $\approx 5 \times 10^{14}$ useful FLOP/s per GPU.
H100-GPU-day: $5 \times 10^{14} \times 86 , 400 \approx 4 . 3 \times 10^{19}$ FLOPs.
Chinchilla-optimal: $N_{T} \approx 20 P$ tokens. Modern runs use $100 P$ - $200 P$ tokens to amortize inference cost (see chapter 14).
MFU target: $\geq 0 . 4$ for well-tuned distributed runs; $< 0 . 1$ signals a system-level bottleneck (communication, data stall, or fragmentation).
Attention quadratic term: adds $\sim B T^{2} / d$ relative to the linear terms; double $T$ adds $4 \times$ attention FLOPs but only $2 \times$ MLP FLOPs.

Caveats

The $6 N_{T} P$ approximation assumes MLP matrix multiplications dominate. Small $d$ or long $T$ can push attention FLOPs to the same order.
Gradient checkpointing re-runs a partial forward pass during the backward, multiplying $C_{\text{fwd}}$ by $\approx 1 . 33$ while saving activation memory.
MFU degrades rapidly under heavy communication (tensor parallelism all-reduces, pipeline bubbles); measure on the actual cluster before committing to a layout.
FP8 training theoretically doubles peak FLOP/s (H100: $\approx$ 1979 TFLOP/s dense); treat it as an advanced option requiring careful numerical validation.

A.3 KV-Cache Sizing and Latency Estimates

Generation is memory-bandwidth-limited: every optimization targets bytes transferred, not FLOPs. The arithmetic intensity argument is in subsection 8.2.1; the full latency-throughput model is in subsection 15.2.2; KV-reduction architectures (GQA, MLA, local attention) are in chapter 16.

Formulas

KV cache per sequence (bf16; $N_{\mathrm{kv}}$ key-value heads; head dimension $H$ ; context $S$ tokens):

C_{\mathrm{kv}} = 2 \times L \times N_{\mathrm{kv}} \times H \times S \times 2 \text{bytes}

(First factor of 2: K and V tensors; last factor of 2: bytes per bf16 element.)

Weight memory (bf16):

M_{\text{weights}} = 2 P \text{bytes}

Decode step time at batch size $B$ (memory-bandwidth-limited):

t_{\text{step}} ( B ) = \frac{M_{\text{weights}} + B \cdot C_{\mathrm{kv}}}{\mathrm{BW}}

Decode throughput (tokens per second):

\mathrm{TPS} ( B ) = \frac{B \cdot \mathrm{BW}}{M \qquad + B \cdot C_{\mathrm{kv}}}

$\text{weights}$ KV-crossover batch size (where KV cache memory $=$ weight memory):

B^{\ast} = \frac{M_{\text{weights}}}{C_{\mathrm{kv}}}

Throughput gains are superlinear in $B$ below $B^{\ast}$ and sublinear above it.

Maximum batch size (VRAM-limited):

B_{\max} = \frac{\mathrm{VRAM} - M_{\text{weights}}}{C_{\mathrm{kv}}}

GQA with $N_{\mathrm{kv}}$ heads (down from $N$ in MHA):

C_{\mathrm{kv}}^{\mathrm{GQA}} = C_{\mathrm{kv}}^{\mathrm{MHA}} \times \frac{N_{\mathrm{kv}}}{N} , \qquad B^{\ast , \mathrm{GQA}} = B^{\ast , \mathrm{MHA}} \times \frac{N}{N_{\mathrm{kv}}}

Worked Example - Llama-2 13B on a Single H100

Config: P = 13 B, L = 40, N = 40, N_kv = 40, H = 128
        BW = 3.35 TB/s, VRAM = 80 GB, S = 1024, bf16

M_weights = 2 * 13e9 = 26 GB
C_kv      = 2 * 40 * 40 * 128 * 1024 * 2 bytes = 838 MB ~ 0.84 GB/seq

B*        = 26 / 0.84 ~ 31
B_max     = (80 - 26) / 0.84 ~ 64  (barely fits)

 B  | Total mem (GB) | Step time (ms) | TPS (tok/s)  | Fits?
 1  |   26.8         |    8.0         |    125       | yes
16  |   39.4         |   11.8         |  1,356       | yes
32  |   52.9         |   15.8         |  2,025       | yes
64  |   79.8         |   23.8         |  2,689       | barely
128 |  133.5         |   39.8         |  3,215       | no (> 80 GB)

With GQA, N_kv = 8 (Llama-3 style; reduction factor = 40/8 = 5):
  C_kv  = 0.84 * (8/40) = 0.168 GB
  B*    = 26 / 0.168 ~ 155
  B_max = (80 - 26) / 0.168 ~ 321

  B = 128: total mem = 26 + 128 * 0.168 = 47.5 GB  (fits)
           step time = 47.5 / 3350 = 14.2 ms
           TPS       = 128 / 0.0142 ~ 9,000 tok/s   (~3.3x vs. MHA at B=64)

Rules of Thumb

Single-sequence latency ( $B = 1$ ): $\approx 2 P [$ GB $] /$ BW $[$ GB/s $]$ ms per token. Llama-2 13 B on H100: $26 / 3 , 350 \approx 8$ ms.
Crossover batch $B^{\ast}$ : where throughput gains shift from superlinear to sublinear. Compute it first; batching beyond $B^{\ast}$ yields diminishing returns.
GQA production default: $N_{\mathrm{kv}} = N / 8$ (group size 8) shrinks $C_{\mathrm{kv}}$ by the reduction factor with $< 1 \%$ accuracy loss at $\geq$ 7 B scale.
KV quantization (int8): halves $C_{\mathrm{kv}}$ , doubling $B_{\max}$ at the same VRAM; use before adding hardware.
Doubling context $S$ : doubles $C_{\mathrm{kv}}$ , halves $B_{\max}$ . At $S = 4 , 096$ , $B^{\ast}$ for Llama-2 13 B drops from $\sim$ 31 to $\sim$ 8.
Replica parallelism: $M$ independent copies give $M \times$ throughput with zero extra latency and no interGPU communication; prefer it over larger batches.

Caveats

These estimates apply to the decode (generation) phase only. Prefill is compute-bound, not memory-bound; see subsection 15.1.1.
Attention during decode has arithmetic intensity $\approx 1$ independent of batch size, because each sequence owns its own KV cache. See subsection 15.1.2.
Inter-GPU communication (tensor/pipeline parallelism) is excluded from the step-time formula; add allreduce latency when sharding the model across multiple devices.
Flash Attention reduces HBM traffic during prefill but does not alter the decode formula above-the KV cache must still be streamed fully from HBM at each generation step.