Notation and Symbols

This chapter standardizes the symbols, abbreviations, and conventions used throughout the book. Definitions are precise; where a symbol carries multiple meanings in the literature, the book’s chosen convention is stated explicitly. Units are always written out on first use in each chapter; the table below lists the canonical forms.

Model and Architecture

Symbol	Meaning	Notes
$P$	Total parameter count	e.g. $P = 7 \times 10^{9}$ for a 7B model
$L$	Number of transformer layers (depth)
$d$	Hidden / residual stream dimension	also written $d_{\text{model}}$
$d_{\text{model}}$	Hidden dimension (explicit form)	synonym for $d$
$d_{\text{head}}$	Attention head dimension	typically 128
$d_{\text{ff}}$	Feed-forward (MLP) intermediate dimension	$\lfloor 8 d / 3 \rfloor$ for SwiGLU; $4 d$ for ReLU/GeLU
$N$	Number of query attention heads
$N_{\text{kv}}$	Number of key-value heads	$N_{\text{kv}} = N$ (MHA); $N_{\text{kv}} < N$ (GQA/MQA)
$H$	Head dimension; also $d_{\text{head}}$	context disambiguates
$H_{Q}$	Query head count	used in GQA ratio $H_{Q} / H_{K V}$
$H_{K V}$	Key-value head count	$H_{K V} = 1$ (MQA); $H_{K V} = H_{Q} / G$ (GQA)
$G$	GQA group size ( $N / N_{\text{kv}}$ )	reduction factor for KV cache
$V$	Vocabulary size	tokens in the tokenizer
$S$	Sequence length in tokens	also written $T$ in some chapters
$T$	Sequence length in tokens	synonym for $S$ ; also used for training tokens
$K_{e}$	Expert utilization rate in MoE	fraction of experts activated per token
$r$	LoRA rank	dimension of low-rank adapters

Training and Optimization

Symbol	Meaning	Notes
$C$	Total compute budget (FLOPs)
$C_{\text{fwd}}$	FLOPs for one forward pass	$\approx 2 N_{T} P$
$C_{\text{step}}$	FLOPs for one training step	$\approx 6 N_{T} P$ (fwd + bwd)
$N_{T}$	Total tokens in a batch: $B \times T$
$D$	Training dataset size (tokens)
$D^{\ast}$	Chinchilla-optimal token count	$D^{\ast} \approx 20 P$ at compute-optimal
$N^{\ast}$	Chinchilla-optimal parameter count
$\eta$	Learning rate
$\alpha$	Adam $\beta_{1}$ decay (first moment)	or generic scaling coefficient
$\beta$	Adam $\beta_{2}$ decay (second moment); also KL penalty coefficient	context disambiguates
$\epsilon$	Adam numerical stability term; also PPO clip ratio tolerance
$m_{t}$	Adam first moment (mean) at step $t$
$v_{t}$	Adam second moment (variance) at step $t$
$\mu$	Momentum coefficient; also mean of a distribution
$\gamma$	Gradient clipping threshold; also discount factor in RL
$\lambda$	L2 regularization weight; also GAE discount in RL

Inference and Serving

Symbol	Meaning	Notes
$B$	Batch size (number of concurrent sequences)
$B_{\max}$	Maximum batch size given VRAM	$\lfloor ( M_{\text{GPU}} - M_{\text{weights}} ) / C_{\text{kv}} \rfloor$
$B^{\ast}$	KV-crossover batch size	$M_{\text{weights}} / C_{\text{kv}}$ ; throughput shifts from superlinear to sublinear
$C_{\text{kv}}$	KV cache memory per sequence	$2 \times L \times N_{\text{kv}} \times H \times S \times 2$ bytes (bf16)
$C_{\text{seq}}$	KV cache memory per sequence (alternate notation)	synonym for $C_{\text{kv}}$
$M_{\text{weights}}$	Model weight memory	$2 P$ bytes in bf16
$M_{\text{GPU}}$	Total GPU VRAM	e.g. 80 GB for H100 SXM
$t_{\text{step}}$	Decode step latency	$( M_{\text{weights}} + B \cdot C_{\text{kv}} ) /$ BW
TPS	Tokens per second (throughput)	aggregate across all concurrent sequences
TTFT	Time to first token	prefill latency; compute-bound
TBT	Time between tokens (decode latency)	per-token; memory-bandwidth-bound
TPOT	Time per output token	synonym for TBT
ITL	Inter-token latency	synonym for TBT / TPOT
$\pi_{\theta}$	Policy (language model) parameterized by $\theta$	used in RLHF/RL chapters
$\pi_{\text{ref}}$	Reference policy (frozen pre-RLHF model)
$r_{\phi}$	Reward model parameterized by $\phi$
$\theta$	Model parameters (generic)
$G$	Number of rollouts per prompt in GRPO	distinct from GQA group size $G$ ; context disambiguates
$\hat{A}_{i}$	Group-normalized advantage for response $i$	GRPO: $( r_{i} - \mu_{R} ) / \sigma_{R}$
$y_{w} , y_{l}$	Preferred / rejected response in a preference pair	DPO/RLHF notation

Hardware and Performance

Symbol	Meaning	Notes
BW	Memory bandwidth (HBM)	GB/s or TB/s
MFU	Model FLOPs Utilization	achieved FLOP/s $\div$ peak FLOP/s; target $\geq 0 . 4$
MBU	Memory Bandwidth Utilization	achieved BW $\div$ peak BW
$P_{\text{attn}}$	Parameters in attention layers	$\approx 4 d^{2}$ per layer (MHA)
$P_{\text{mlp}}$	Parameters in MLP layers	$3 d d_{\text{ff}}$ per layer (SwiGLU)
$B_{r} , B_{c}$	FlashAttention tile (block) sizes for rows and columns	SRAM tile dimensions
$A_{k} , B_{k}$	Left and right matrix tiles in tiled GEMM
$t_{\text{wave}}$	Time to execute one SM wave	GPU scheduling unit
$N_{\text{total}}$	Total expert count in MoE
$N_{\text{active}}$	Active experts per token in MoE	$N_{\text{active}} \ll N_{\text{total}}$

Alignment and Post-Training

Symbol	Meaning	Notes
$D_{\text{KL}} ( \cdot \\| \cdot )$	Kullback-Leibler divergence
$D_{\text{KL}} ( \pi_{\theta} \\| \pi_{\text{ref}} )$	Policy drift from reference	regularization term in RLHF/DPO
$\sigma$	Standard deviation; also sigmoid activation	context disambiguates
$p , q$	Generic probability distributions
$x$	Input token sequence
$i , j , k$	Generic indices

Common Abbreviations

Term	Meaning
MHA	Multi-Head Attention
GQA	Grouped-Query Attention
MQA	Multi-Query Attention (GQA with $N_{\text{kv}} = 1$ )
MLA	Multi-Head Latent Attention
KV cache	Key-Value cache stored between decode steps
HBM	High-Bandwidth Memory (GPU DRAM)
SRAM	Static RAM; on-chip shared memory / L1 cache
SM	Streaming Multiprocessor (GPU compute unit)
FLOP	Floating-point operation (one multiply-add $= 2$ FLOPs)
MFU	Model FLOPs Utilization
TP	Tensor Parallelism (intra-layer, intra-node)
PP	Pipeline Parallelism (inter-layer, inter-node)
DP	Data Parallelism
ZeRO	Zero Redundancy Optimizer (stages 1-3)
FSDP	Fully Sharded Data Parallelism (ZeRO stage 3 in PyTorch)
SFT	Supervised Fine-Tuning
RLHF	Reinforcement Learning from Human Feedback
DPO	Direct Preference Optimization
PPO	Proximal Policy Optimization
GRPO	Group Relative Policy Optimization
RLVR	Reinforcement Learning from Verifiable Rewards
LoRA	Low-Rank Adaptation
QLoRA	Quantized LoRA
PEFT	Parameter-Efficient Fine-Tuning
MoE	Mixture of Experts
CoT	Chain of Thought
BF16	Brain Float 16 (1 sign, 8 exponent, 7 mantissa bits)
FP16	IEEE Float 16 (1 sign, 5 exponent, 10 mantissa bits)
INT8	8-bit integer quantization
SLO	Service Level Objective
QPS	Queries per second

Conventions

Tokens vs. sequences. “Tokens” refers to individual vocabulary elements; “sequence” or “context” refers to an ordered list of tokens. Batch size $B$ counts sequences, not tokens; $N_{T} = B \times T$ counts tokens.
FLOPs counting. One multiply-accumulate (MAC) operation $= 2$ FLOPs. The formula $C_{\text{fwd}} \approx 2 N_{T} P$ follows this convention. “FLOP/s” (floating-point operations per second) uses the same factor.
Memory units. $1$ GB $= 10^{9}$ bytes throughout (SI prefix, not binary). Bandwidth is reported in GB/s or TB/s.
Latency units. Milliseconds (ms) for per-token and per-request latencies; seconds (s) for end-to-end generation time; microseconds ( $\mu$ s) for kernel-level timings.
Throughput units. Tokens per second (tok/s or tokens/s) for generation throughput; queries per second (QPS) for request-level throughput.
Precision notation. Memory footprints assume bf16 (2 bytes/parameter) unless otherwise stated. The training-memory figure of $16 P$ bytes assumes AdamW in fp32.
Prefill vs. decode. Prefill encodes the full prompt in one parallel forward pass (compute-bound). Decode generates tokens one at a time in an autoregressive loop (memory-bandwidth-bound). These two phases have distinct bottlenecks and optimizations.
Latency vs. throughput. TTFT optimizes differently from aggregate TPS. Minimize TTFT with small batches and fast prefill hardware; maximize TPS with large batches and high memory bandwidth. A single serving configuration cannot simultaneously optimize both without workload-specific scheduling.
Overloaded symbols. $G$ denotes both the GQA group size and the GRPO rollout count; $T$ denotes both sequence length and training token count; $\beta$ denotes both the Adam second-moment decay and the KL penalty coefficient. In each case, context and the surrounding equation make the intended meaning unambiguous.