Chapter 31 Designing a Fine-Tuning and Alignment Pipeline

31.1 Scoping the Problem

31.1.1 Choosing Between SFT, DPO, and RLVR

When an interviewer asks you to design a post-training pipeline from scratch, the very first decision separates candidates who have shipped these systems from those who have only read about them: which optimization objective do you reach for? Supervised fine-tuning, DPO, and RL from verifiable rewards (RLVR) are not interchangeable-each one assumes a different answer to the question of what signal you have and what behavior you are trying to unlock. This question appears at senior and staff levels at every frontier lab. Getting the choice wrong at scoping time wastes months of compute and annotator budget before a single evaluation number tells you something went sideways.

All three methods share the same goal: given a pre-trained base model $\pi_{\text{base}}$ , produce a policy $\pi_{\theta}$ that behaves better on your target task. They differ in what “better” means and how that signal is delivered.

SFT treats post-training as distribution matching. You collect a set $\mathcal{D} = \{ ( x_{i} , y_{i}^{\ast} ) \}$ of prompt-completion pairs from experts or high-quality sources and minimize:

\mathcal{L}_{\text{SFT}} ( \theta ) = - \sum_{i} \log \pi_{\theta} ( y_{i}^{\ast} | x_{i} )

The model learns to imitate the conditional distribution over demonstrations. SFT is powerful at teaching output style-if you fine-tune on responses with structured lists and clear explanations, you reliably get that format back. But it does not reliably teach new knowledge. If the SFT target $y^{\ast}$ references facts the base model does not already encode, the model learns to reproduce the structural pattern of a knowledgeable response without the substance, opening a systematic hallucination pathway.

DPO sidesteps expert demonstrations entirely. You collect pairwise preferences: for each prompt $x$ , a chosen completion $y_{w}$ and a rejected completion $y_{l}$ . The InstructGPT RLHF framework models these preferences via the Bradley-Terry equation $P ( y_{w} \succ y_{l} | x ) = \sigma ( r ( x , y_{w} ) - r ( x , y_{l} ) )$ . DPO’s key insight is that under a KL-regularized RL objective, the optimal policy implies a reward expressible purely in terms of log-ratios. Substituting this reparameterization into the Bradley-Terry loss and canceling the intractable partition function gives:

\mathcal{L}_{\text{DPO}} ( \theta ) = - \mathbb{E} \biggl[ \log \sigma \biggl( \beta \log \frac{\pi_{\theta} ( y_{w} | x )}{\pi_{\text{ref}} ( y_{w} | x )} - \beta \log \frac{\pi_{\theta} ( y_{l} | x )}{\pi_{\text{ref}} ( y_{l} | x )} \biggr) \biggr]

This is a supervised objective: increase the log-probability of $y_{w}$ relative to the reference, decrease that of $y_{l}$ , with gradient magnitude scaling up when the implied reward ordering disagrees with the human label.

RLVR abandons the pairwise structure entirely. A reward function $r ( x , y ) \in \{ 0 , 1 \}$ checks objective correctness-does the final answer match the ground truth? GRPO, the most practical algorithm for this setting, samples a group of $G$ completions $\{ y_{1} , . . . , y_{G} \}$ from the same prompt and defines each completion’s advantage as the group-normalized z-score of its reward:

\hat{A}_{i} = \frac{r_{i} - \mu_{r}}{\sigma_{r} + \epsilon} , \qquad \mu_{r} = \frac{1}{G} \sum_{j = 1}^{G} r_{j} , \sigma_{r} = \sqrt{\frac{1}{G} \sum_{j = 1}^{G} ( r_{j} - \mu_{r} )^{2}}

The policy gradient update clips the likelihood ratio $\rho_{i} = \pi_{\theta} ( y_{i} | x ) / \pi_{\text{old}} ( y_{i} | x )$ inside $[ 1 - \epsilon , 1 + \epsilon ]$ and adds a per-token KL penalty against the reference policy. Critically, there is no separate value network: the group mean serves as the variance-reduction baseline, cutting memory cost roughly in half compared to PPO.

SFT teaches what good outputs look like; DPO teaches which outputs humans prefer; RLVR teaches the model to actually solve problems it can be graded on.

SFT’s strength is speed and stability. There is no reward hacking, no rollout overhead, and the training dynamics are identical to pre-training. Its ceiling, however, is the quality of demonstrations. If your annotators are more knowledgeable than your model, you risk teaching structural imitation rather than knowledge retrieval. The model learns: whenever I see a complex question I should output a long, citation-bearing answer-even when the citation is fabricated.

DPO moves above that ceiling because pairwise comparisons are faster to collect and humans can often judge which response is better even when they cannot produce the best response themselves-the generatorvalidator gap. The weakness is overoptimization. With enough gradient steps on a fixed preference dataset, the reward implied by the policy diverges from true human preferences. Preference data collected from one checkpoint does not remain informative as the policy drifts, producing responses that ace the proxy loss but disappoint on fresh human evaluation.

RLVR sidesteps the overoptimization problem by using objective, automatable rewards that cannot be gamed by exploiting annotator inconsistency. You can optimize substantially further before hitting diminishing returns. The cost is domain restriction: RLVR only works where automated correctness checking is cheap and unambiguous-math, code, formal logic. Open-ended creative writing or nuanced safety judgments do not satisfy this requirement. The practical consequence is that most production pipelines layer all three in sequence: SFT first to establish behavioral format, DPO for broad helpfulness and safety alignment, RLVR on top for specific verifiable task domains.

A Worked Example

Suppose you are building a coding assistant on a 7B base model (32 layers, $d = 4096$ , bf16 precision, fits on a single H100 with room for a $2 \times$ context). Pass@1 on HumanEval starts at 32%.

SFT stage. Collect 50,000 prompt-completion pairs of coding problems with step-by-step solutions. Train for 2 epochs at a learning rate of $2 \times 10^{- 5}$ with a cosine schedule-roughly 4 hours on 8 H100s. After SFT, the model reliably wraps code in fenced blocks, obeys instructions like “add type hints,” and handles multi-turn context. pass@1 climbs to 61%.

DPO stage. Run 200,000 pairwise comparisons: for each coding prompt, generate two completions from the SFT model and have GPT-4 label which is more correct and readable. Train one DPO epoch at $\beta = 0 . 1$ . pass@1 reaches 67% and human preference win-rate versus the SFT checkpoint reaches 58%. Values of $\beta$ below 0.05 produce reward-hacked outputs; above 0.5 the policy barely moves from the reference.

RLVR stage. Construct 4,000 LeetCode-style problems with deterministic test suites. For each training step, sample $G = 8$ completions per prompt, execute them against the test harness, and assign $r \in \{ 0 , 1 \}$ . Run GRPO for 500 steps with $\text{clip} \epsilon = 0 . 2$ and KL coefficient $\beta_{\text{KL}} = 0 . 04$ . pass@1 on HumanEval reaches 76%. On competition-level problems never seen during SFT or DPO, pass@1 jumps from 11% to 29%-a capability gain that imitation-based methods cannot reach.

What You Actually Decide in Practice

SFT before any RL, always. Skipping SFT forces RL to learn output format and task behavior simultaneously, wasting early steps and producing unstable training curves. Even 1,000 high-quality demonstrations are enough to prime the format before RLVR takes over.
Keep SFT data within the base model’s knowledge horizon. If the base model cannot generate a fact chain in its top-200 samples, that knowledge is not reliably stored. Feeding it SFT examples that depend on that fact teaches hallucination, not retrieval. Curate or ablate: either verify the model already knows the fact or replace the answer with an explicit abstention.
DPO’s $\beta$ lives in $[ 0 . 05 , 0 . 5 ]$ in practice. Sweep it on a held-out preference win-rate evaluation before a full run. Below 0.05 you get incoherent reward-hacked outputs within two epochs; above 0.5 you pay the cost of DPO data collection for nearly zero policy change.
RLVR requires executability, not just labels. If evaluating one rollout takes more than a few hundred milliseconds, training will be GPU-bound on evaluation latency rather than compute. Domains that satisfy the speed requirement: Python test harness for code ( $\sim$ 100ms per rollout), SymPy or LLM equivalence checker for math ( $\sim$ 50ms). Domains that do not: human safety review, long-form factuality audits.
Remove length normalization from GRPO. Dividing rewards by response length inside the advantage calculation-present in the original GRPO formulation-incentivizes incorrect completions to become arbitrarily long, since length deflates the penalty. Drop this term; control output length separately via a length reward applied only after initial RL convergence, once the model is already finding correct solutions.

How This Shows Up in Interviews

“Walk me through how you would choose between SFT, DPO, and GRPO for a new product feature.” Strong answer: map the feature onto three axes-do you have expert demonstrations, human pairwise preferences, or automatable correctness signals?-then layer them in order (SFT $\rightarrow$ DPO $\rightarrow$ RLVR) only as far as signal quality and compute budget allow. If you have none of the above, collect SFT data first.
“Why would DPO underperform PPO in some experimental setups?” Strong answer: DPO is inherently offline-it fixes the preference dataset at collection time and does not update rollouts as the policy shifts. Once the policy drifts from the generating distribution, implied rewards grow stale and gradients mislead. PPO’s on-policy rollouts keep the gradient signal current. Iterated DPO-re-collecting preferences from the evolving policy each round-closes this gap but costs as much as PPO to run.
Senior follow-up: “DeepSeek R1 found that process reward models did not outperform outcomebased rewards. Why, and what does that imply for your pipeline design?” Senior answer: process reward models require dense, step-level supervision that is expensive to collect and prone to step-level reward hacking-the model learns to satisfy the PRM’s grading rubric without actually solving the problem. Outcome rewards are sparser but directly optimize the terminal objective, and at sufficient rollout scale the model receives adequate gradient signal from the binary outcome alone. The practical implication: build outcome-based RLVR first; add a PRM only if outcome rewards plateau and the domain supports reliable step-level annotation. Even then, use the PRM as a dense shaping signal added to the outcome reward, not as a replacement for it.

Key Takeaways

SFT reliably imprints output style via distribution matching but cannot push beyond demonstrated capability; it is the mandatory first stage before any RL, even when RL is the ultimate objective.
DPO converts pairwise preferences into a supervised loss by reparameterizing reward as a log-ratio of policies; it is cheaper than SFT data collection but susceptible to overoptimization as the policy drifts from the preference-collection distribution.
RLVR via GRPO unlocks capability gains beyond human imitation using group-normalized outcome rewards and no value model, but requires domains where correctness is automatable and evaluation is fast.
The production recipe is additive and ordered: SFT establishes format, DPO aligns general preferences, RLVR squeezes task-specific performance-run each stage only as far as your signal quality and latency budget allow.
The first question when scoping a post-training pipeline is never “which algorithm?”-it is “what labels do I have, at what cost, and how fast can I evaluate them?”

31.1.2 Defining Rewards and Evaluation Harnesses

Once you have chosen an algorithm, the next thing that will kill your project is a poorly designed reward or an evaluation harness that does not measure what you think it measures. In my experience this is where most post-training pipelines fail: not in the optimizer or the parallelism strategy, but in the silent divergence between the proxy signal you can compute cheaply and the actual behavior you want to ship. Interviewers at senior and staff levels probe this directly-“how would you evaluate whether the model actually improved?” is one of the most loaded questions in the interview, because a weak answer reveals that you have only thought about training, not about whether training is doing the right thing.

Every post-training objective requires a reward signal $r ( x , y ) \in \mathbb{R}$ that scores a completion $y$ against a prompt $x$ . The design of this function-and the harness used to measure it at evaluation time-determines whether your training loop is converging toward the behavior you want or toward a proxy that happens to correlate with it in your current data distribution.

Rewards decompose into three practical categories:

Outcome rewards assign a single scalar to the complete response based on whether it achieves a verifiable end state. For code, this is test-suite pass rate. For math, it is final-answer equivalence checked by SymPy or an LLM judge. For factual QA with known answers, it is exact or normalized string match. The signal is binary or near-binary: $r \in \{ 0 , 1 \}$ . Outcome rewards are the backbone of RLVR pipelines because they are fast, automatable, and resistant to gaming-the model cannot satisfy a test suite by producing a plausible-looking answer that fails on edge cases.

Process rewards assign intermediate scores to individual reasoning steps. A process reward model (PRM) reads a chain-of-thought token-by-token and emits a scalar at each step or at step boundaries, giving the RL algorithm dense feedback rather than a single terminal signal. This is appealing in theory because sparse terminal rewards are hard to optimize over long sequences. In practice, PRMs are expensive to build-you need step-level human annotations or a synthetic verifier that is itself reliable-and they introduce a second reward-hacking surface: the model learns to satisfy the PRM’s grading criteria at each step without the overall reasoning being correct.

Preference rewards are the output of a reward model (RM) trained on pairwise human annotations. Given a prompt $x$ and two completions $y_{w} \succ y_{l}$ , the RM minimizes a Bradley-Terry cross-entropy loss:

\mathcal{L}_{\text{RM}} ( \phi ) = - \mathbb{E} \bigl[ \log \sigma ( r_{\phi} ( x , y_{w} ) - r_{\phi} ( x , y_{l} ) ) \bigr]

The trained $r_{\phi}$ then scores any completion as a scalar during RL. Preference rewards are the only practical option for open-ended domains where no automatic verifier exists. Their weakness is the overoptimization cliff: as the RL policy moves away from the preference-collection distribution, the RM’s scores become unreliable, and further optimization maximizes the proxy without improving-or actively degrading-true human preference.

An evaluation harness is the broader infrastructure that runs a fixed set of prompts through a checkpoint, computes the reward (or a separate held-out metric), and returns a scalar summary. The harness is distinct from the training reward. You should never evaluate on the exact reward function used during training, because that would conflate proxy optimization with actual improvement.

The training reward is a signal you optimize; the evaluation harness is the honest answer to whether that signal pointed in the right direction.

The central tension in reward design is between measurability and alignment. Outcome rewards are maximally measurable-a unit test either passes or fails-but they only cover domains with ground-truth answers. Preference rewards are broadly applicable but soft, noisy, and gameable. Most production pipelines combine them: outcome rewards for hard skills where correctness is unambiguous, preference rewards for general helpfulness and safety, and a separate held-out evaluation harness for both.

The overoptimization phenomenon is the clearest argument for treating the evaluation harness as a first-class artifact rather than an afterthought. In a representative study, models optimized against a preference RM showed increasing RM score throughout training but declining true human preference win-rate after a critical point-proxy and reality diverged. The same curve appears for every noisy reward: the model finds the cracks in the measurement before you do. Building a harness that is harder to optimize than the training reward-human evals sampled from a held-out prompt distribution, or a separate RM trained on a disjoint annotation set-gives you an early-warning system.

For verifiable rewards, the evaluation harness should test generalization, not coverage. If you trained on 4,000 LeetCode-style problems, your harness should run on a disjoint set of 500 problems from a different difficulty distribution, including problems harder than anything in the training set. Pass rate on the training distribution tells you the model memorized; pass rate on the out-of-distribution harness tells you it learned.

A Worked Example

Consider designing the reward stack for a math reasoning model fine-tuned from a 7B base.

Training reward. Binary outcome: the model generates a response inside <think> and </think> tags followed by a final boxed answer. A lightweight SymPy equivalence checker compares the extracted answer to the ground-truth numeric or symbolic value. Runtime per rollout: approximately 40 ms. Format compliance (presence of the required tags) contributes a secondary reward of $+ 0 . 1$ to prevent the model from abandoning the structured output early in training. The two rewards are summed: $r = r_{\text{correct}} + 0 . 1 \cdot r_{\text{format}}$ .

Evaluation harness. Four held-out benchmark splits, none overlapping with training data:

MATH-500: 500 competition math problems across five difficulty levels. Reports pass@1 as the primary capability metric.
AIME 2024: 30 problems from the most recent American Invitational Mathematics Examination. Tests extreme difficulty generalization; a well-trained 7B should reach 15-25%.
GSM8K: 1,319 grade-school math problems. A sanity check that RL has not catastrophically degraded basic arithmetic competence; score should remain above 85%.
Overrefusal probe: 100 benign prompts that use mathematical vocabulary but request no reasoning (e.g., “How many days are in a week?”). Checks that the model does not wrap trivial answers in unnecessary long chains-of-thought, which inflates inference cost.

Run the harness every 50 gradient steps. Plot MATH-500 pass@1 and AIME pass@1 together. If MATH- 500 climbs but AIME is flat, the model is pattern-matching training problems rather than learning to reason. If GSM8K drops below 80%, the KL coefficient is too small and capability regression has set in-increase $\beta_{\text{KL}}$ and restart from the last good checkpoint.

What You Actually Decide in Practice

Never evaluate on the training reward function directly. Use a structurally different evaluator-a stricter equivalence checker, a separate RM trained on held-out annotations, or human preference votes on a disjoint prompt sample. Evaluating on the training reward confirms the optimizer converged; it says nothing about whether behavior improved.
Instrument for length separately from quality. Length is the most common confound in post-training evaluation. Human judges and LLM judges both show a 60-70% preference for longer responses regardless of quality. Log response length at every harness run and check that quality gains are not simply length gains. Normalize pass rates by length bucket when comparing checkpoints.
Set a regression floor for every capability you are not actively training. If you are RL-ing on math, your code benchmark and factual QA score should be checked at every harness run. RLHF and RLVR both cause capability regression when the KL penalty is too weak; the harness regression floor catches this before the checkpoint is promoted.
For preference rewards, log the RM score distribution, not just the mean. Overoptimization shows up first in the shape of the distribution: the model learns to score in a narrow band near the RM’s saturation point, producing stylistically homogeneous but semantically thin responses. A narrowing standard deviation of RM scores-even when the mean is rising-is an early warning sign.
Match the evaluation prompt distribution to the deployment prompt distribution. A harness built from academic benchmarks will not catch failures on the prompts your users actually send. If you have production traffic, sample 1,000 prompts per week, have them human-rated, and add them to the harness. This is the highest-signal investment in evaluation infrastructure you can make.

How This Shows Up in Interviews

“How would you evaluate whether your post-training run actually improved the model?” Strong answer: distinguish the training reward (proxy, optimized) from the evaluation harness (held-out, harder to game), name at least two harness components-a capability benchmark and a regression floor-and mention the length confound and overoptimization cliff as things you are actively watching.
“Your RLHF reward model score is going up every epoch but human raters say the model is getting worse. What is happening?” Strong answer: overoptimization-the policy has drifted outside the distribution on which the RM was trained, and the RM is now unreliable. Mitigations: add a KL penalty to slow policy drift, collect fresh preference data from the current policy (on-policy DPO), or switch to an outcome reward in a verifiable sub-domain to anchor the signal.
Senior follow-up: “How would you design a reward for a general-purpose assistant where there is no single correct answer?” Senior answer: layer a weak verifiable signal (format compliance, citation presence, tool-call correctness) with a preference RM for open-ended quality, and track both separately in the harness. Use constitutional AI-style LLM judging for specific safety dimensions rather than a monolithic RM, so you can diagnose which dimension is degrading. Treat the evaluation harness as a multi-objective Pareto surface: you want to see Pareto improvement across all dimensions with each checkpoint, not a single-number summary that masks trade-offs.

Key Takeaways

Rewards split into three types-outcome (binary, automatable, resistant to gaming), process (dense, expensive to build, second hacking surface), and preference (broadly applicable, noisy, subject to overoptimization)- and most production pipelines combine all three.
The evaluation harness is structurally separate from the training reward: it must be harder to optimize, drawn from a held-out distribution, and include at least one regression floor for each capability domain not being actively trained.
Overoptimization is the dominant failure mode in preference-reward pipelines; it shows up as rising RM scores alongside falling true win-rate, and the first detectable signal is narrowing of the RM score distribution before the mean turns over.
Length is a universal confound-normalize quality metrics by response length and log length separately at every harness evaluation.
The most durable evaluation investment is sampling from real production traffic weekly and adding those human-rated examples to the harness; no academic benchmark matches the distribution of prompts your users actually send.

31.2 Operational Concerns

31.2.1 Rollout Infrastructure and vLLM Integration

When you sit down to design an RL training system for a reasoning model, the question that trips most candidates is not algorithmic - it is operational. Specifically: where does inference happen, who manages the weights, and how do you keep GPU utilization from collapsing while rollouts are in flight? This topic shows up in senior and staff-level system design interviews precisely because it sits at the intersection of distributed training and inference serving, two domains that most engineers treat as separate concerns.

The RL training loop for language models has two structurally different phases per iteration. The first is rollout generation: autoregressively sampling responses from the current policy for every prompt in the batch. This is pure inference - it saturates memory bandwidth rather than compute, benefits from continuous batching and paged attention, and produces variable-length outputs. The second is the parameter update: a forward pass to recompute log probabilities, a backward pass to accumulate gradients, and an optimizer step. This workload needs the full model in fp32 or bf16 with optimizer state alongside it, and it needs the entire activation graph in memory during the backward pass.

These two workloads cannot share the same GPU allocation efficiently. The rollout phase needs large KV-cache buffers; the training phase needs activation memory. Running both sequentially on one device forces you to deallocate and reallocate large memory regions each iteration, fragmenting the allocator and throttling throughput. The standard architecture therefore separates them into dedicated worker groups:

Rollout workers: run the current policy via vLLM or a similar engine; produce (prompt, response, log_prob) tuples.
Trainer workers: hold model weights plus optimizer state; run forward-backward-optimizer steps.

The data flow per outer iteration is:

Trainer
  → broadcast weights to rollout workers (NCCL)
  → rollout workers generate responses + log probs
  → responses + log probs sent back to trainer
  → trainer runs N inner gradient steps
  → repeat

The rollout worker is an inference server that the trainer treats as an external data source; weight synchronization is the only coupling between them.

Separating the two workloads lets each worker optimize for its own regime. The rollout worker can use continuous batching to maximize KV-cache utilization: shorter sequences finish early and free slots for new ones without stalling the rest. The trainer can plan memory layout for activation checkpointing without worrying about KV-cache pages. The two run on different devices, so their memory pressure does not interfere.

The main trade-off is weight synchronization latency. After every outer iteration, the trainer must push updated weights to all rollout workers before the next generation pass starts. With NCCL collective broadcasts this takes on the order of seconds for a 7B model over NVLink, but it stalls the rollout workers during the transfer. Without overlapping this transfer with the optimizer step on the trainer side, synchronization becomes the wall-clock bottleneck.

The obvious alternative - a single process that interleaves inference and training calls - works in toy implementations but fails at scale. Memory fragmentation from alternating large allocation patterns (KV cache versus activation buffers) degrades throughput, and there is no opportunity to overlap computation across the two phases.

A Worked Example

Consider a 7B model in bf16, running GRPO on a cluster of H100s (80 GB HBM each). Model weights: $7 \times 10^{9} \times 2$ bytes $\approx 14$ GB. AdamW optimizer state (first and second moments, fp32): $\approx 56$ GB. The trainer worker therefore needs approximately 70 GB just for weights and optimizer state, leaving only 10 GB for activation memory - tight but feasible with gradient checkpointing.

On the rollout worker side, a single H100 running vLLM with 60 GB allocated to KV cache can hold roughly $60 \times 10^{9} / ( 2 \times 32 \times 128 \times 2 ) \approx$ 36,000 key-value slots per layer, supporting batches of 64 sequences at 2K tokens each with room to spare. The rollout worker leaves the remaining 20 GB for the model weights themselves (no optimizer state, no activations beyond the forward pass).

Weight synchronization for 14 GB over a 600 GB/s NVLink interconnect takes about 0.02 seconds - negligible. Over a 200 Gb/s Ethernet fabric between nodes, it takes about 0.56 seconds and dominates iteration time if not overlapped. The standard mitigation is to start the broadcast while the trainer is still computing gradients on the previous batch, so the weights arrive just as the trainer finishes its optimizer step.

Regarding vLLM’s current limitation: as of early 2025, hot-updating weights inside a running vLLM instance requires an experimental NCCL-based API that loads weight tensors from the trainer’s memory space into vLLM’s internal buffers via collective operations. Many production implementations instead tear down and restart vLLM each outer iteration, paying a few seconds of initialization overhead. At batch sizes large enough to amortize that cost, this is acceptable; at small batch sizes, it dominates.

What You Actually Decide in Practice

Rollout reuse vs. freshness: GRPO takes $K$ inner gradient steps per rollout batch to amortize inference cost. Each inner step recomputes log probs under the current policy - this is a cheap forward pass, not a new generation. Never re-run the rollout worker for inner steps; just cache the token sequences and recompute log probs from the trainer model.
Variable-length CoT handling: Long-chain reasoning models generate sequences ranging from 200 to 8K tokens in the same batch. Naive padding to the maximum length wastes $5 \times$ to $40 \times$ memory on a mixed batch. Use sequence packing (concatenate sequences up to the context limit) or length-based bucketing (group sequences within 10% of each other by length) before passing to the trainer.
Reward computation parallelism: If the reward requires executing sandboxed code (Python, unit tests), reward evaluation can take hundreds of milliseconds per sample. Decouple reward evaluation entirely from GPU workers: run it on CPU workers or a separate service that returns scores asynchronously while the trainer is doing inner gradient steps on the previous batch.
Memory for old log probs: Do not store a full copy of $\pi_{\text{old}}$ on GPU to compute clipping ratios. At generation time, record the log probability of each sampled token and store it as a float tensor. For a batch of 256 sequences of 1K tokens, that is $256 \times 1024 \times 4$ bytes $= 1$ GB - two orders of magnitude cheaper than a second model copy.
When to use vLLM weight update vs. restart: Use in-place weight update (NCCL) when your outer iteration time exceeds 60 seconds, so the 2-5 second restart overhead matters. Below that threshold, restart is simpler and avoids subtle bugs with LoRA adapter or quantized layer synchronization that the experimental API does not yet handle cleanly.

How This Shows Up in Interviews

“Walk me through the infrastructure for a GRPO training run at scale.” A strong answer covers: dedicated rollout and trainer workers, one-way weight broadcast per outer iteration, rollout reuse with log-prob recomputation for inner steps, and decoupled reward evaluation. Mention the vLLM integration and the teardown-vs-hot-update trade-off.
“Why is RL harder to scale than supervised fine-tuning?” Strong answer: inference is in the critical path (rollouts must complete before any gradient step), multiple model copies increase peak memory, weight synchronization adds inter-node traffic that does not exist in SFT, and variable-length outputs from autoregressive generation break the uniform batch structure that SFT relies on for high utilization.
Senior follow-up: “How would you overlap rollout generation and gradient computation?” Senior answer: double-buffering. While the trainer runs $K$ inner gradient steps on batch $N$ , the rollout workers are already generating batch $N + 1$ using the weights from iteration $N$ (slightly stale, but within the trust region). The weight broadcast for iteration $N + 1$ is then pipelined to overlap with the last inner step of batch $N$ . This hides almost all synchronization latency but requires careful accounting: log probs for batch $N + 1$ are computed under iteration- $N$ weights, so the old-policy log probs in the clipping ratio must match.

Key Takeaways

Rollout generation and gradient updates have incompatible memory layouts; separate them into dedicated worker groups to avoid fragmentation and enable overlapped execution.
Weight synchronization from trainer to rollout workers is the primary inter-iteration latency; use NCCL broadcasts and overlap them with the optimizer step where possible.
Cache rollout token sequences and old log probs as tensors; never re-run the inference engine for inner gradient steps.
Variable-length CoT outputs require sequence packing or length bucketing; unmitigated padding can waste $10 \times$ or more of memory bandwidth.
Reward evaluation should be decoupled from GPU workers and parallelized on CPU or a dedicated service to avoid blocking the training pipeline.

31.2.2 Keeping Reference, Old, and Current Policies in Sync

One of the most reliable ways to identify an engineer who has implemented GRPO versus one who has only read about it is to ask them how many policy versions they maintain during training and what role each one plays. The bookkeeping here is operationally simple but conceptually precise: get it wrong and you produce gradients that appear to train (the loss decreases) while actually computing the wrong thing. This question appears at the senior-level intersection of RL algorithms and distributed systems correctness.

A GRPO training loop maintains up to three distinct versions of the policy simultaneously, each frozen at a different cadence:

$\pi_{\theta}$ (current policy): the model being trained; weights change after every inner gradient step.
$\pi_{\text{old}}$ (old policy): a snapshot frozen at the start of each outer iteration; used to compute the importance sampling ratio for the PPO-clip objective.
$\pi_{\text{ref}}$ (reference policy): a slowly updated or permanently frozen baseline from the SFT checkpoint; used to compute the KL penalty that prevents catastrophic policy drift.

The clip objective relies on the ratio $r_{t} = \pi_{\theta} ( a_{t} | s_{t} ) / \pi_{\text{old}} ( a_{t} | s_{t} )$ . For the importance weight to be valid, $\pi_{\text{old}}$ must be treated as a constant with respect to gradient computation. The same applies to $\pi_{\text{ref}}$ in the KL term. The critical implementation requirement is:

# WRONG: differentiates through old_log_probs
ratio = torch.exp(log_probs - old_model(responses))

# CORRECT: old log probs are a frozen tensor
with torch.no_grad():
    old_log_probs = compute_log_probs(old_model, responses)
# old_log_probs has no grad_fn; safe to use in loss
ratio = torch.exp(log_probs - old_log_probs)

The KL penalty uses the low-variance estimator rather than the naive sample average:

\hat{D}_{\text{KL}} ( \pi_{\theta} \| \pi_{\text{ref}} ) = \frac{\pi_{\text{ref}}}{\pi_{\theta}} - \log \frac{\pi_{\text{ref}}}{\pi_{\theta}} - 1

This has the same expectation as $\log ( \pi_{\theta} / \pi_{\text{ref}} )$ but lower variance because the $r - \log r - 1$ form eliminates the first-order noise term. The update schedule across the three policy roles is:

$\pi_{\text{ref}} :$ never or every $\mathcal{O} ( 100 )$ outer iterations $\pi_{\text{old}} :$ once per outer iteration $\pi_{\theta} :$ every inner step

The three policy roles exist because each one serves a different contract: $\pi_{\theta}$ is what we optimize, $\pi_{\text{old}}$ is the anchor for off-policy correction, and $\pi_{\text{ref}}$ is the anchor for behavioral stability.

$\pi_{\text{old}}$ enables rollout reuse. Without it, each gradient step would require a new rollout from the current policy, multiplying inference cost by the number of inner steps $K$ . The importance ratio $\pi_{\theta} / \pi_{\text{old}}$ corrects for the distributional shift introduced by doing multiple updates on stale rollouts. The clip constraint $\text{clip} ( r_{t} , 1 - \epsilon , 1 + \epsilon )$ bounds how stale the correction can be before it is simply ignored, preventing large policy updates from producing high-variance gradients.

$\pi_{\text{ref}}$ solves a different problem. Verifiable rewards (math correctness, code passing tests) do not capture general language quality, instruction-following behavior, or formatting. A policy optimized purely for reward can drift into a mode that solves problems correctly but becomes incoherent on other tasks - reward hacking at the behavioral level. The KL penalty against $\pi_{\text{ref}}$ provides a soft tether to the instruction-tuned starting point.

The obvious simplification - pure on-policy REINFORCE with a single policy copy - is mathematically valid but practically inefficient. Every gradient step requires a full rollout, making the cost proportional to $K$ times higher compared to GRPO with rollout reuse. The price of GRPO’s efficiency is the bookkeeping complexity of three policy versions.

A Worked Example

Consider a 7B model in bf16 on a single H100 (80 GB HBM). Weights: 14 GB. AdamW optimizer state (fp32 moments for $\pi_{\theta}$ ): 56 GB. Total for the trainable model: 70 GB. That leaves 10 GB.

Option A: store $\pi_{\text{old}}$ as a full GPU model copy. Cost: 14 GB. Now at 84 GB - over capacity.

Option B: compute old log probs at rollout time and store them as a float32 tensor. For a batch of 256 sequences of 1,024 tokens each:

256 \times 1024 \times 4 \text{bytes} = 1 , 048 , 576 \text{bytes} \approx 1 \text{GB}

Total: 71 GB. Fits with 9 GB to spare for activations.

For $\pi_{\text{ref}}$ , the same 14 GB weight cost applies. Options: (a) keep it on GPU if memory permits, (b) offload to CPU and transfer to GPU only during the KL computation step (once per batch), or (c) quantize $\pi_{\text{ref}}$ to int8 (7 GB) since it never needs gradients. With option (b), the CPU-to-GPU transfer of 14 GB at 50 GB/s (PCIe 4.0) takes about 0.28 seconds; with NVLink on a multi-GPU system this drops to under 0.03 seconds. For outer iterations that take 30+ seconds end-to-end, CPU offloading is essentially free.

What You Actually Decide in Practice

Never store $\pi_{\text{old}}$ as a full model copy on GPU. Compute log probs at rollout time and retain them as a torch.Tensor with requires_grad=False. You need only the scalar log probability per token, not the ability to re-run inference.
Wrap all reference and old log prob computations in torch.no_grad(). This is not optional. If you omit it and $\pi_{\text{old}}$ shares parameters with $\pi_{\theta}$ (e.g., they are the same object), the gradient of $\log ( \pi_{\theta} / \pi_{\text{old}} )$ is identically zero, your optimizer makes no effective update, and the loss still decreases. The bug is completely silent without inspecting gradient norms.
$\pi_{\text{ref}}$ update cadence should be slow. Updating it every outer iteration makes it equivalent to $\pi_{\text{old}}$ , which already serves the off-policy correction role. A common pattern is to fix $\pi_{\text{ref}}$ at the SFT checkpoint for the entire run, or to update it every 200-500 outer iterations as a soft annealing schedule.
Under tensor parallelism, all three policy shards must live on the same device mesh with consistent sharding. If $\pi_{\text{ref}}$ is sharded differently from $\pi_{\theta}$ (e.g., different TP degree), the all-reduce communication during the KL computation will deadlock or produce incorrect partial results. Initialize all three from the same checkpoint and shard them identically.
For DPO, $\pi_{\text{ref}}$ is the only extra policy; there is no $\pi_{\text{old}}$ . The DPO loss computes $\log \pi_{\theta} ( y_{w} | x ) - \log \pi_{\theta} ( y_{l} | x ) - \log \pi_{\text{ref}} ( y_{w} | x ) + \log \pi_{\text{ref}} ( y_{l} | x )$ . The reference log probs can be precomputed once on the entire dataset and stored as a scalar per example, reducing the runtime memory footprint to a lookup table rather than a live model forward pass.

How This Shows Up in Interviews

“Describe the three policy roles in GRPO and when each one updates.” A strong answer names all three explicitly: $\pi_{\theta}$ updates every inner step; $\pi_{\text{old}}$ updates once per outer iteration (after rollout generation); $\pi_{\text{ref}}$ is fixed or updated very rarely. Each role is tied to its function: optimization target, importance weight anchor, behavioral stability anchor.
“What goes wrong if you accidentally differentiate through $\pi_{\text{old}}$ ?” Strong answer: when $\pi_{\text{old}}$ shares the same parameter tensor as $\pi_{\theta}$ , the ratio $\pi_{\theta} / \pi_{\text{old}}$ is a constant 1 regardless of the weights. Its gradient with respect to $\theta$ is zero by the chain rule, so the clipping term contributes nothing to the parameter update. The KL term may still produce a nonzero gradient, so training appears to proceed, but the clip mechanism is broken and the trust region guarantee is gone.
Senior follow-up: “How do you manage the three policies under model-parallel training across 64 GPUs?” Senior answer: all three policy copies must use identical tensor-parallel and pipeline-parallel sharding configurations. For $\pi_{\text{ref}}$ , since it never needs a backward pass, you can quantize it to int8 and co-locate it on the same device mesh as $\pi_{\theta}$ without a full fp16 copy, halving its HBM footprint. For $\pi_{\text{old}}$ , avoid storing model weights at all: at rollout time, the rollout worker (which runs $\pi_{\text{old}}$ by definition, since weights were just synchronized) records log probs and returns them to the trainer as a flat tensor. The trainer stores this tensor; no second model shard is needed.

Key Takeaways

Three policy roles: $\pi_{\theta}$ (trained every inner step), $\pi_{\text{old}}$ (frozen per outer iteration for importance sampling), $\pi_{\text{ref}}$ (frozen or slowly updated for KL regularization).
Store $\pi_{\text{old}}$ as a log-prob tensor, not a model copy; at 7B scale this saves roughly 13 GB of HBM per policy copy.
Missing torch.no_grad() on old or reference log prob computation produces a silent zero-gradient bug that is invisible from the loss curve alone.
$\pi_{\text{ref}}$ update cadence must be slow; frequent updates collapse it to $\pi_{\text{old}}$ and eliminate the behavioral stability anchor.
Under model parallelism, all three policy shards must use identical device mesh and sharding configuration to avoid communication deadlocks.