Chapter 27 Preference-Based Alignment (RLHF)

27.1 The RLHF Pipeline

27.1.1 Pairwise Preference Collection

You are asked in a system design interview to improve a model that has already been fine-tuned on expert demonstrations. The interviewer wants to know what comes next. The canonical answer is reinforcement learning from human feedback-and the first question you will face is where the feedback comes from. Pairwise preference collection is the annotation backbone of the entire RLHF pipeline: the mechanism that converts vague human preferences into a learnable scalar signal. This topic appears consistently at senior ML engineer and research scientist interviews at any lab running a post-training pipeline, and the ability to reason about annotation quality, cost trade-offs, and systematic biases separates a strong answer from a shallow one.

The fundamental shift from supervised fine-tuning to RLHF is a shift in what you ask humans to do. In SFT you ask: write the ideal completion. In RLHF you ask: which of these two completions is better? The first question demands generation; the second demands only discrimination. Humans are systematically better at the second task-they can reliably judge quality even when they cannot produce it themselves under time pressure or without deep domain expertise.

The standard setup is straightforward. Given a prompt $x$ , the current model generates $k \geq 2$ candidate responses $\{ y_{1} , . . . , y_{k} \}$ . An annotator reviews pairs $( y_{A} , y_{B} )$ and marks a preference: A is better, B is better, or roughly equal. Each labeled pair becomes training signal for a reward model. The InstructGPT annotation guidelines operationalized this with three axes to judge: helpfulness, truthfulness, and harmlessness.

The optimization objective that consumes these preferences is:

\max_{\pi_{\theta}} \mathbb{E}_{x \sim \mathcal{D} , y \sim \pi_{\theta} ( y | x )} \bigl[ r_{\phi} ( x , y ) \bigr] - \beta \mathrm{KL} \bigl[ \pi_{\theta} ( y | x ) \| \pi_{\mathrm{SFT}} ( y | x ) \bigr]

The first term maximizes the learned reward; the KL penalty keeps the RL policy from drifting too far from the SFT initialization, suppressing reward hacking and preserving fluent language generation. In practice $\beta$ sits in the range $0 . 01$ - $0 . 1$ ; larger values produce more conservative, SFT-like outputs.

Pairwise feedback exploits the generator-validator gap: humans validate far more reliably than they generate, and RLHF collects supervision by asking the easier of the two questions.

SFT has a structural bottleneck: demonstration quality is bounded by what annotators can produce under real-world constraints. When you ask a crowd to write completions for a nuanced question about Bayesian inference or macroeconomic policy, you get short, hedged, often incomplete responses-because producing expert-level prose in five minutes under pay-per-task conditions is genuinely hard. You are paying for the full cognitive cost of generation.

Pairwise comparison costs a fraction of that. The same annotator who would spend fifteen minutes writing a mediocre demonstration can evaluate two strong model-generated responses in two to three minutes and give you a reliable ranking. Empirically, inter-annotator agreement on pairwise helpfulness judgments runs around 70-80%, which is comparable to the agreement you get from a strong AI judge. That signal-cheaper to collect and often more consistent than demonstrations on the same budget-is what reward model training runs on.

The trade-off is information density. A demonstration gives you a full token-level trajectory; a pairwise comparison gives you one bit of relative signal. You compensate with volume: InstructGPT trained its reward model on roughly 50,000 pairwise comparisons while using only about 13,000 SFT demonstrations. Because the model itself generates both responses, on-policy rollouts produce new pairs at near-zero marginal cost once the infrastructure is in place.

A Worked Example

Suppose you are building an RLHF annotation pipeline for a 7B chat model and you have a $50,000 annotation budget. Expert annotators cost roughly $25 per hour.

Option A-SFT demonstrations: a high-quality multi-paragraph demonstration takes approximately twenty minutes to write, costing about $8.33 per example. Your budget yields roughly 6,000 demonstrations.

Option B-pairwise feedback: judging a pair takes approximately five minutes, costing about $2.08 per comparison. The same budget yields roughly 24,000 comparisons. The model itself generates both responses, so annotation cost does not scale with response length.

The quality picture is even starker. In my experience watching large-scale annotation projects, speed constraints destroy signal quality for anything requiring verification. Reports from a production assistant’s annotation guidelines described a target rate of approximately one judgment per minute per question. At that pace an annotator must read two responses of several hundred words each, compare them on helpfulness and safety, and check the relevant box-leaving no time to verify embedded citations or check arithmetic. Studies of crowdworker behavior under speed pressure confirm what you would expect: annotators correlate strongly with formatting signals (bullet-point density, paragraph count, word count) and weakly with factual accuracy.

Consider the summarization case: expert writers asked to compare their own summaries to model-generated ones showed a consistent preference for the AI outputs. One explanation collected in a follow-up interview: “When I wrote, I felt pressure to use flowery language. The AI ones just read better.” Generation and evaluation are different cognitive tasks, and people are often better at the latter-even when judging their own work.

What You Actually Decide in Practice

Human vs. AI feedback. GPT-4-class models agree with human consensus at roughly 70-75% on capability judgments-on par with human inter-annotator agreement-at orders of magnitude lower cost. For most helpfulness and instruction-following axes, AI feedback is the production default today. Reserve human annotation for high-stakes safety and factual domains where a rubric cannot fully capture the judgment required.
Control for length bias explicitly. Both humans and AI judges prefer longer responses at a 60-70% base rate, independent of quality. If you ignore this, your reward model learns to produce verbose outputs rather than correct ones. Fix it by including length-controlled pairs in your annotation set, by adding an explicit conciseness criterion to the rubric, or by normalizing response length during reward model training.
On-policy data is more valuable than off-policy data for reward model training. On-policy pairswhere both responses come from the model you are currently optimizing-anchor the reward model to the policy distribution you actually care about. Off-policy data from unrelated models adds distributional coverage early in training but becomes stale quickly as your model improves. Refreshing pairwise datasets every major training iteration is the standard pattern at labs running continuous post-training.
Annotator demographics shape model values. RLHF sits at the end of the pipeline and has outsized influence on surface behavior. When the annotator pool skews toward a particular nationality or cultural background, the resulting model shows measurably stronger alignment with that group’s norms-detectable in controlled probing studies. Audit annotator composition before deployment, especially on topics where cultural context affects what counts as helpful or harmful.
AI self-preference is a real confound in evals. When you use a model to judge its own outputs versus a competitor’s, it exhibits strong self-preference at rates that do not correspond to human rankings. Always use a judge model that is distinct from the model under evaluation; otherwise your win-rate numbers are systematically inflated.

How This Shows Up in Interviews

“Why does RLHF outperform SFT for instruction following at scale?” Strong answer: SFT is bounded by what annotators can generate under time and cost pressure; pairwise preference collection exploits the generator-validator gap, extracting more usable signal per annotation dollar. The KL divergence penalty in the training objective keeps the RL policy grounded in the SFT distribution, preventing the degenerate outputs that unconstrained reward maximization produces.
“What are the main failure modes of pairwise preference collection?” Strong answer: length bias (60- 70% preference for longer responses regardless of quality), inability to verify facts under speed pressure, AI judge self-preference bias, and annotator demographic skew that embeds cultural assumptions into the reward model.
Senior follow-up: “How do you detect whether your reward model has learned real quality or just superficial style?” Strong answer: hold out a set of pairwise examples where ground truth is independently verifiable-math problems with known answers, code with executable test suites, factual QA with reference sources. Measure reward model ranking accuracy on this set. If RM scores correlate more strongly with response length than with ground-truth correctness, the model has overfit to style. Fix it by augmenting the pairwise training set with verified correct-versus-incorrect pairs, adding explicit accuracy and conciseness dimensions to the annotation rubric, and-where possible-using verifiable rewards (correct math, passing tests) to anchor the preference signal.

Key Takeaways

Pairwise feedback exploits the generator-validator gap: judging two responses requires a fraction of the time and expertise needed to generate a high-quality demonstration from scratch, which is why the same annotation budget buys roughly four times as many pairwise comparisons as SFT examples.
The InstructGPT training objective couples reward maximization with a KL penalty against the SFT model, keeping the RL policy from drifting into degenerate, reward-hacking outputs.
AI judges at the GPT-4 class match human inter-annotator agreement at roughly 70-75% on most capability axes and are the production default at scale today; human annotation is reserved for high-stakes safety and factual verification tasks.
Length bias is a systematic confound: both humans and AI judges show a 60-70% base preference for longer responses independent of quality; failing to control for it trains a reward model that optimizes verbosity.
On-policy pairwise data-pairs generated by the model currently under optimization-is more valuable than off-policy data and should be refreshed regularly as the model improves.

27.1.2 Reward Modeling and the Bradley-Terry Assumption

You have collected tens of thousands of pairwise preference labels and you need to turn them into a signal that a neural network can optimize against. The interviewer asks: “How does a collection of binary A-vs-B comparisons become a reward function?” The answer is a probabilistic assumption called the Bradley-Terry model and a reward model trained by maximum likelihood on that assumption. This topic comes up at MLE and research scientist interviews whenever RLHF is on the table, and the depth of your answer-whether you can derive the loss from first principles, name its failure modes, and explain when you would skip the reward model entirely-is what separates candidates at the senior and staff levels.

The Bradley-Terry model posits a latent, scalar-valued ground truth: every possible response $y$ to a prompt $x$ has a true scalar reward $r ( x , y ) \in \mathbb{R}$ . We never observe $r$ directly. What we observe is noisy pairwise comparisons generated according to:

P ( y_{w} \succ y_{l} | x ) = \sigma \bigl( r ( x , y_{w} ) - r ( x , y_{l} ) \bigr)

where $\sigma$ is the logistic sigmoid, $y_{w}$ is the preferred (“winner”) response, and $y_{l}$ is the less preferred (“loser”) response. The probability of preferring $y_{w}$ is a logistic function of the difference in their latent rewards. If $r ( x , y_{w} )$ exceeds $r ( x , y_{l} )$ by a large margin, the preference is nearly deterministic; if the two rewards are equal, the annotator is effectively flipping a coin.

Given a dataset $\mathcal{D} = \{ ( x_{i} , y_{w}^{i} , y_{l}^{i} ) \}_{i = 1}^{N}$ of pairwise labels, training a reward model $r_{\phi}$ means maximizing the log-likelihood of the observed preferences under this model. The negative log-likelihood loss is:

\mathcal{L} ( \phi ) = - \frac{1}{N} \sum_{i = 1}^{N} \log \sigma \bigl( r_{\phi} ( x_{i} , y_{w}^{i} ) - r_{\phi} ( x_{i} , y_{l}^{i} ) \bigr)

Minimizing $\mathcal{L}$ pushes $r_{\phi} ( x , y_{w} )$ above $r_{\phi} ( x , y_{l} )$ for every observed preferred pair-exactly the ranking we want.

In practice, $r_{\phi}$ is implemented by taking a pre-trained language model, discarding the vocabulary projection head, and adding a single linear layer that maps the final hidden state of the last token to a scalar. The resulting network reads the concatenation of prompt and response and outputs one number: the estimated quality of that response.

The reward model is the bridge between human preference data and an optimization landscape: it converts a discrete ranking over pairs into a continuous, differentiable signal that the RL training loop can ascend.

The Bradley-Terry assumption is the minimal probabilistic model that is consistent with pairwise preference data. It has one free parameter per sequence-the latent reward-and generates binary comparisons via a logistic noise model. That simplicity is a feature: the loss function is convex in the reward differences and well-understood, and the model generalizes beyond the observed pairs because $r_{\phi}$ is parameterized by a neural network that shares weights across all prompts and responses.

The key advantage over alternatives is coverage. If you trained only on SFT demonstrations you would have no way to score outputs the model generates at RL time-the reward model fills that gap, acting as a proxy judge for any $( x , y )$ pair the policy might produce, including ones never seen during data collection.

The critical trade-off is that the reward model is a proxy, not the true objective. It is trained on a finite sample of human comparisons, which means it has blind spots: regions of output space where it assigns high reward to responses that humans would reject. The standard mitigation is the KL penalty in the RL objective, which keeps the policy close enough to the SFT distribution that the reward model’s estimates remain reliable. When the policy strays too far from that distribution-a phenomenon called reward overoptimization-RM scores continue rising while actual human preference scores plateau and eventually fall.

A Worked Example

Consider training a reward model for a 7B chat model. You have 50,000 pairwise comparisons collected on 10,000 distinct prompts (roughly five comparisons per prompt).

Architecture. Start from the 7B SFT checkpoint. Replace the final unembedding matrix (vocabulary dimension $\sim$ 32,000) with a single linear layer mapping from the model’s hidden dimension ( $d = 4 , 096$ ) to a scalar. Total added parameters: 4,096. The remaining 7B parameters are fine-tuned on the reward modeling loss.

Batch construction. Each training example is a triplet $( x , y_{w} , y_{l} )$ . Both $y_{w}$ and $y_{l}$ are concatenated with $x$ and forward-passed through the same network in a single batch. You read off the scalar at the last token position for each, then compute $\sigma ( r_{\phi} ( x , y_{w} ) - r_{\phi} ( x , y_{l} ) )$ and take the log. A batch of 64 pairs requires two forward passes of the 7B model, so effective batch size for memory purposes is 128 sequences.

Numbers. With 50,000 pairs, a learning rate of $1 \times 10^{- 5}$ , and 1-2 epochs of training on an H100 (roughly 80 GB HBM), reward model training typically completes in a few hours. The resulting RM achieves human preference accuracy (does the RM agree with the human annotator on held-out pairs?) of around 65-72% on general helpfulness tasks and lower on specialized factual domains-which is the quantitative signal you use to decide whether the RM is good enough to drive RL training.

Calibration check. Evaluate RM score versus ground-truth preference on a held-out factual QA set where correct answers are known. If Spearman correlation between RM score and factual accuracy is below 0.5, the RM is primarily scoring style and the RL run will optimize verbosity rather than correctness.

What You Actually Decide in Practice

Reward model size relative to policy size. The RM does not need to be as large as the policy. In production pipelines I have seen 3B RMs driving 70B RL runs effectively. A smaller RM trains faster and is easier to refresh as new preference data arrives. The floor is that the RM must be large enough to understand the semantic content it is scoring; going below roughly one quarter of the policy size tends to hurt quality noticeably.
Margin-based loss variants. The standard Bradley-Terry loss treats all pairs equally regardless of how clear the human preference was. Many teams add an annotation confidence score and train with a margin term:

\mathcal{L} = - \log \sigma \bigl( r_{\phi} ( x , y_{w} ) - r_{\phi} ( x , y_{l} ) - m \bigr)

where $m > 0$ is a margin that forces the RM to score winning responses by at least $m$ above losing ones. This sharpens the decision boundary and reduces reward hacking at the cost of slightly lower recall on ambiguous pairs.

When to skip the explicit reward model (DPO). If your dataset is fixed and you do not need to run on-policy rollouts, Direct Preference Optimization eliminates the RM entirely by reparameterizing the reward in terms of the policy’s own log-probability ratio. DPO is simpler to implement and avoids RM training costs. Use it when your pairwise data is off-policy and relatively small ( $\lesssim$ 100K pairs). Use an explicit RM when you need on-policy rollouts at RL time-the RM must score live outputs that do not exist in any static dataset.
Distribution shift is the main failure mode. The RM is calibrated on responses similar to those in the pairwise dataset. As RL training proceeds, the policy shifts and its outputs diverge from the RM’s training distribution. Monitor RM score versus human evaluation score on a rolling held-out set throughout RL training; if they diverge, either stop training, refresh the RM with on-policy preference data, or tighten the KL coefficient $\beta$ .
Ensemble or leave-one-out calibration for high-stakes deployment. A single RM can overfit to annotation artifacts. Training two or three RMs from different random seeds and using their minimum score (rather than mean) as the optimization target is a practical hedge against reward overoptimization in production settings where you cannot run frequent human evals.

How This Shows Up in Interviews

“Walk me through how you train a reward model from pairwise preferences.” Strong answer: start with the Bradley-Terry model-each response has a latent scalar reward, preferences are logistic in the reward difference-then write the negative log-likelihood loss over preferred and rejected pairs. Implement as a pretrained LM with the vocab head replaced by a scalar linear layer. Train with learning rate around $10^{- 5}$ for 1-2 epochs; evaluate with held-out pairwise accuracy.
“What can go wrong with reward model training and how do you detect it?” Strong answer: the two main failure modes are length bias (RM scores longer responses higher regardless of quality) and distribution shift during RL (RM extrapolates incorrectly to outputs it was not trained on). Detect length bias by measuring correlation of RM scores with response length on a neutral prompt set; detect distribution shift by tracking the gap between RM score and human eval score as RL training progresses.
Senior follow-up: “When would you choose DPO over training an explicit reward model?” Strong answer: DPO reparameterizes the reward in terms of the policy’s own log-ratio, eliminating the RM and the RL training loop. It is the right choice when the pairwise dataset is static (you will not collect on-policy preferences), the policy is moderate in size (under $\sim$ 30B), and you want a simpler pipeline. Use an explicit RM plus PPO when you need on-policy rollouts-live scoring of model-generated outputs that do not exist in any precomputed dataset-or when your RM needs to be shared across multiple policy variants.

Key Takeaways

The Bradley-Terry model assumes every response has a latent scalar reward; pairwise preferences are logistic in the reward difference, giving a clean maximum-likelihood training objective: minimize $- \log \sigma ( r_{\phi} ( x , y_{w} ) - r_{\phi} ( x , y_{l} ) )$ over all preference pairs.
The reward model is architecturally a pretrained LM with its vocabulary head replaced by a single scalar linear layer; it is trained on the same pairwise data and produces a continuous quality estimate for any $( x , y )$ the policy might generate.
The RM is a proxy, not the true objective: it generalizes poorly to outputs far from its training distribution, so the KL penalty in the RL objective is essential for keeping the policy in-distribution for the RM.
The primary failure mode is reward overoptimization: RM scores rise while true human preference scores plateau or fall; monitor with a rolling held-out human eval throughout RL training and refresh the RM when divergence appears.
When pairwise data is static and on-policy rollouts are not needed, DPO eliminates the explicit RM by reparameterizing reward in terms of the policy’s own log-probability ratio, reducing the pipeline to a single supervised training run.

27.2 Optimization Algorithms

27.2.1 PPO for Language Models

You are asked in an interview to design the optimization loop that turns a reward model into a better language model policy. The moment you say “gradient descent on the reward,” the follow-up is: “what exactly is your loss, and why do you need a value network?” PPO is the answer InstructGPT gave in 2022, and frontier labs continued using variants of it long after DPO arrived. Understanding PPO concretely-not just as a named algorithm but as a sequence of decisions-is the minimum bar for an RLHF deep-dive at the staff-engineer level.

The goal is to find a policy $\pi_{\theta}$ that maximizes expected reward while staying close to the supervised fine-tuned reference model $\pi_{\text{ref}}$ . The InstructGPT objective formalizes this as:

\max_{\pi_{\theta}} \mathbb{E}_{x \sim \mathcal{D} , y \sim \pi_{\theta} ( \cdot | x )} [ r_{\phi} ( x , y ) ] - \beta D_{\mathrm{KL}} ( \pi_{\theta} ( \cdot | x ) \| \pi_{\text{ref}} ( \cdot | x ) )

The KL term keeps the RL policy from drifting so far from $\pi_{\text{ref}}$ that it produces fluent-sounding but rewardhacked outputs. The scalar $\beta$ is a regularization coefficient; typical values range from 0.01 to 0.1.

Starting from the plain policy gradient, the gradient of $\mathbb{E}_{\pi_{\theta}} [ r ( z ) ]$ with respect to $\theta$ equals $\mathbb{E}_{\pi_{\theta}} [ r ( z ) \nabla \log_{\theta} \pi_{\theta} ( z ) ]$ . This is REINFORCE: multiply the log-probability gradient by the reward and update accordingly. It is unbiased but high-variance. Two refinements turn it into PPO.

First, replace the raw reward with an advantage $A_{t}$ -the reward minus a learned state-value baseline $V_{\phi} ( s_{t} )$ . Subtracting any baseline that does not depend on the current action leaves the gradient unbiased while dramatically reducing variance. In practice $A_{t}$ is computed via Generalized Advantage Estimation (GAE) with hyperparameters $\lambda$ and $\gamma$ ; setting both to 1 collapses GAE to a simple return-minus-baseline form, which already works well.

Second, allow multiple gradient steps on a single set of rollouts using importance-weighted corrections. If the current policy is $\pi_{\theta}$ but samples came from the older $\pi_{\theta_{\text{old}}}$ , the corrected objective becomes:

\mathcal{L}^{\mathrm{CLIP}} ( \theta ) = \mathbb{E} [ \min_{t} ( \rho_{t} A_{t} , \operatorname{clip} ( \rho_{t} , 1 - \epsilon , 1 + \epsilon ) A_{t} ) ] , \rho_{t} = \frac{\pi_{\theta} ( a_{t} | s_{t} )}{\pi_{\theta_{\text{old}}} ( a_{t} | s_{t} )}

The clip removes any incentive to push the probability ratio outside $[ 1 - \epsilon , 1 + \epsilon ]$ ; a typical value is $\epsilon = 0 . 2$ . This is the PPO-Clip objective-no KL constraint to tune, just a hard floor and ceiling on how much the policy can shift per update.

PPO is policy gradient with variance reduction via a value baseline, importance weighting for multi-step updates, and a clipping constraint that naturally keeps the new policy close to the old one.

The plain policy gradient fails at scale because its gradient variance grows with sequence length-a 512- token response has 512 token-level log-probabilities whose products create enormous variance. Subtracting the value baseline brings that variance down to something tractable without introducing bias.

The clipping mechanism does something subtle: it makes the optimization landscape less adversarial. Without clipping, a lucky update can send the policy into a region of high reward-model scores but low actual quality. Clipping hard-caps how far a single update can travel, which is especially important in the first few thousand steps when the reward model is most at risk of being exploited.

Compared to the alternative of using a hard KL constraint (TRPO), PPO is much simpler to implement and tune. TRPO requires a second-order computation to enforce the constraint; PPO replaces that with a clip and a step-size schedule. The performance tradeoff is minimal, which is why PPO absorbed TRPO’s market share almost entirely after its 2017 release.

The cost is real, though: PPO for language models requires four live models-the policy, the reference policy, the reward model, and the value model. The value model is typically as large as the policy, so GPU memory roughly doubles compared to SFT.

A Worked Example

Consider a 7B parameter policy (e.g. Llama-3-7B) in bf16 on 8 $\times$ H100 80 GB GPUs, with a 7B reward model and a 7B value model.

Memory budget. Each 7B model in bf16 uses roughly 14 GB for weights. With activations and optimizer state, a single model consumes around 60-80 GB during a forward-backward pass. Running policy $+$ reference $+$ reward $+$ value simultaneously requires memory-efficient strategies: the reference and reward models are frozen (no gradients), so they can be kept in inference mode at $\approx$ 14 GB each. Total active memory: $\sim$ 14 (ref) $+$ 14 (reward) $+$ 70 (value, with optimizer) $+$ 70 (policy, with optimizer) $\approx$ 168 GB, which fits across 8 $\times$ 80 GB with tensor parallelism.

One PPO step, end to end.

Rollout. Sample $B = 64$ prompts from the dataset; run the policy autoregressively to generate responses of up to 512 tokens. This is the slow step: at 5,000 tokens/s on an H100 with vLLM, 64 sequences of 512 tokens takes $\sim$ 6.5 seconds.
Reward and value computation. Feed each (prompt, response) pair through the reward model to get scalar $r ( x , y )$ . Feed each token through the value model to get $V_{\phi} ( s_{t} )$ at every position.
KL penalty. Compute per-token KL: $\mathrm{kl}_{t} = \log \pi_{\theta} ( a_{t} | s_{t} ) - \log \pi_{\text{ref}} ( a_{t} | s_{t} )$ . Add $- \beta \cdot \mathrm{kl}_{t}$ to the terminal reward to build a per-token reward signal.
GAE. With $\gamma = 1$ , $\lambda = 0 . 95$ , compute advantages $A_{t}$ backward from the last token.
Policy update. Run $K = 4$ mini-batch gradient steps on $\mathcal{L}^{\mathrm{CLIP}}$ with $\epsilon = 0 . 2$ . Monitor the mean ratio $\rho_{t}$ ; if it drifts above 1.3 consistently, reduce learning rate or decrease $K$ .
Value update. Run a parallel MSE loss on the value head to keep $V_{\phi} ( s_{t} )$ calibrated.

In practice, 1,000 such PPO steps on 64K training prompts takes 6-8 hours on this hardware. Reward usually peaks and plateaus within the first 500 steps; continuing further without increasing data diversity leads to overoptimization.

What You Actually Decide in Practice

$\beta$ , the KL coefficient. Start at 0.04-0.1. If you see the KL divergence from the SFT model exceeding 10 nats after a few hundred steps, increase $\beta$ . If reward barely moves, lower it. Some implementations anneal $\beta$ during training.
Number of PPO epochs per rollout batch ( $K$ ). Common choices are 1-4. More epochs extract more signal per rollout, but stale importance weights cause the clip to activate more and gradients become increasingly biased. At $K = 1$ you are essentially doing online policy gradient without any staleness; $K = 4$ requires careful monitoring.
Value model initialization. Always initialize the value model from the same checkpoint as the policy, not randomly. A poorly initialized value model produces high-variance advantages for the first thousands of steps and can destabilize training. Add a warm-up phase of pure value training on rollouts before joint optimization if instability appears.
Reward shaping: per-token vs. terminal. The true task reward (“did the model answer correctly?”) is a terminal reward on the last token. The KL penalty is applied per token. Never add the terminal reward to every token-this creates a heavily biased gradient that overweights early tokens in the response. Assign terminal reward only at the EOS token position.
Memory vs. throughput tradeoff. If GPU memory is the bottleneck, offload the frozen reference and reward models to CPU between rollout batches. This trades $\sim$ 30% throughput for halving active GPU memory. Alternatively, quantize the reference and reward models to int8 (they are frozen, so QAT is unnecessary).

How This Shows Up in Interviews

“Walk me through the PPO loop for a language model.” Strong answer: describe the four models, the rollout step, reward computation, KL penalty construction, GAE advantage estimation, and the clipped surrogate loss. Name the clip range $\epsilon$ and the KL coefficient $\beta$ with representative numbers. Many candidates skip the value model or the KL shaping entirely.
“Why can’t you just do gradient ascent directly on the reward model output?” Strong answer: reward models are finite-capacity proxies; direct gradient ascent exploits their blind spots (overoptimization). The KL term and the clip prevent the policy from moving too far into high-rewardmodel-score but low-true-quality regions. Without these constraints, you get reward hacking within hundreds of steps.
“What is the value model for in PPO, and can you remove it?” Strong answer: the value model provides a variance-reducing baseline for the policy gradient. Without it, gradient variance scales with sequence length and training is unstable. You can remove it, and GRPO does exactly this by replacing the value baseline with a group mean reward, at the cost of requiring multiple rollouts per prompt to compute that baseline.
Senior follow-up: “PPO for an LLM is actually a contextual bandit, not a full MDP. What changes?” In a contextual bandit, there are no intermediate state transitions: the prompt is the context, the full response is the action, and the reward arrives once. This means the credit assignment problem simplifiesyou do not need multi-step Bellman backups. The consequence is that $\gamma$ can safely be set to 1 without worrying about discounting future states, and the value function only needs to estimate the expected return of the entire completion rather than a trajectory. The KL penalty still matters as a regularizer, but it is applied token-by-token as a reward shaping term rather than as a constraint over state transitions. Recognizing the bandit structure is why GRPO can discard the value network entirely and use a withingroup mean as a baseline without loss of correctness.

Key Takeaways

PPO requires four live models during training: policy, reference (frozen), reward (frozen), and value. The value model is the primary memory and complexity cost; its role is variance reduction via advantage estimation.
The clip range $\epsilon = 0 . 2$ prevents any single update from moving the policy too far from its previous state, replacing the second-order KL constraint of TRPO with a simpler and cheaper mechanism.
In the language-model setting, the reward is a terminal scalar; the KL divergence from the reference policy is applied per token as a shaped per-step penalty, not as a hard constraint.
Reward peaks and then plateaus or declines as the policy begins to exploit the reward model’s weaknesses; monitoring KL divergence and true human preference win-rates together is the only reliable way to catch overoptimization early.
GRPO eliminates the value model by computing a group-mean baseline over multiple rollouts of the same prompt, making it the practical default when memory and implementation complexity are constraints.

27.2.2 DPO and Its Derivation

At some point in any RLHF system design discussion, the interviewer will ask why you need PPO at all. The correct answer is: you often do not. Direct Preference Optimization (DPO) removes the reward model, the value model, and the RL loop entirely, collapsing the RLHF objective into a supervised cross-entropy loss over preference pairs. It is the algorithm behind nearly every open-weight RLHF release between mid-2023 and late-2024, and understanding its derivation is what separates a candidate who has “used DPO” from one who actually understands it.

Start from the same KL-regularized reward objective as PPO:

\max_{\pi_{\theta}} \mathbb{E}_{x \sim \mathcal{D} , y \sim \pi_{\theta} ( \cdot | x )} [ r ( x , y ) ] - \beta D_{\mathrm{KL}} ( \pi_{\theta} ( \cdot | x ) \| \pi_{\text{ref}} ( \cdot | x ) )

The key insight in DPO is to ask: if we made no parametric assumptions and allowed $\pi_{\theta}$ to be any function, what would the optimal policy look like? Working through the calculus of variations, the closed-form optimum is:

\pi^{\ast} ( y | x ) = \frac{1}{Z ( x )} \pi_{\text{ref}} ( y | x ) \exp \biggl( \frac{1}{\beta} r ( x , y ) \biggr)

where $Z ( x ) = \sum_{y} \pi_{\text{ref}} ( y | x ) \exp ( r ( x , y ) / \beta )$ is a partition function. Solving for the reward gives:

r ( x , y ) = \beta \log \frac{\pi^{\ast} ( y | x )}{\pi_{\text{ref}} ( y | x )} + \beta \log Z ( x )

The partition function $\log Z ( x )$ depends only on $x$ , not on $y$ . Plug this expression for $r$ into the BradleyTerry preference probability $p ( y_{w} \succ y_{l} | x ) = \sigma ( r ( x , y_{w} ) - r ( x , y_{l} ) )$ . The $\log Z ( x )$ terms cancel, and the preference probability becomes:

p ( y_{w} \succ y_{l} | x ) = \sigma \biggl( \beta \log \frac{\pi_{\theta} ( y_{w} | x )}{\pi_{\text{ref}} ( y_{w} | x )} - \beta \log \frac{\pi_{\theta} ( y_{l} | x )}{\pi_{\text{ref}} ( y_{l} | x )} \biggr)

Maximizing the log-likelihood of the observed pairwise preferences yields the DPO loss:

\mathcal{L}_{\mathrm{DPO}} ( \theta ) = - \mathbb{E}_{( x , y_{w} , y_{l} ) \sim \mathcal{D}} \biggl[ \log \sigma \biggl( \beta \log \frac{\pi_{\theta} ( y_{w} | x )}{\pi_{\text{ref}} ( y_{w} | x )} - \beta \log \frac{\pi_{\theta} ( y_{l} | x )}{\pi_{\text{ref}} ( y_{l} | x )} \biggr) \biggr]

There is no reward model parameter to train, no value network, and no rollout loop. Both $\pi_{\theta}$ (the policy being trained) and $\pi_{\text{ref}}$ (the frozen SFT checkpoint) are evaluated on the already-collected preferred and rejected completions, making every gradient step a standard forward-backward pass.

DPO turns the RL problem into supervised learning by showing that the optimal policy and the reward function are two parametrizations of the same object, so you can optimize preference likelihood directly without ever representing the reward explicitly.

PPO requires on-policy rollouts: you must sample from your current policy, evaluate those samples with the reward model, and update. DPO is fully offline: it consumes a static dataset of (prompt, chosen, rejected) triples. This eliminates the slowest part of RLHF-autoregressive generation at training time-and removes the need for a reward model and a value model, halving the GPU memory requirement compared to PPO.

The DPO gradient has an interpretable form. Let $\hat{r}_{\theta} ( x , y ) = \beta \log ( \pi_{\theta} ( y | x ) / \pi_{\text{ref}} ( y | x ) )$ be the implicit reward. The gradient upweights the chosen completion and downweights the rejected completion, with a weight equal to $\sigma ( \hat{r}_{\theta} ( y_{l} ) - \hat{r}_{\theta} ( y_{w} ) )$ -the probability that the model currently gets the pair wrong. When the model already ranks the pair correctly (weight near 0), the gradient is small; updates concentrate on hard examples. This is analogous to focal loss in classification.

The tradeoff vs. PPO is that DPO is offline: it never generates new completions to explore the reward landscape. If the reference policy $\pi_{\text{ref}}$ never produced diverse enough completions during dataset construction, DPO cannot escape that blind spot. Online variants (iterative DPO, RLHF-Flow) address this by periodically regenerating the rejected set from the current policy, recovering much of PPO’s on-policy advantage while keeping the simpler loss.

A Worked Example

Setup: a 7B model (Mistral-7B-v0.1) fine-tuned with SFT to produce $\pi_{\text{ref}}$ , now trained with DPO on the UltraFeedback dataset (60K preference pairs), in bf16 on 4 $\times$ A100 80 GB GPUs.

Memory. The policy $\pi_{\theta}$ requires $\sim$ 70 GB with optimizer states (AdamW + gradient buffers). The reference $\pi_{\text{ref}}$ is frozen and kept in bf16 inference mode: $\sim$ 14 GB. Total: $\sim$ 84 GB across 4 $\times$ 80 GB GPUs with no tensor parallelism needed-compare to PPO’s $>$ 160 GB.

One DPO gradient step.

Assume a mini-batch of 4 triples $( x_{i} , y_{w , i} , y_{l , i} )$ with max sequence length 512.

Forward pass $\pi_{\theta}$ on all 8 sequences ( $y_{w}$ and $y_{l}$ concatenated with their prompts): compute token log-probabilities, sum over response tokens only to get $\log \pi_{\theta} ( y_{w} | x )$ and $\log \pi_{\theta} ( y_{l} | x )$ .
Forward pass $\pi_{\text{ref}}$ on the same 8 sequences (no gradient): get $\log \pi_{\text{ref}} ( y_{w} | x )$ and $\log \pi_{\text{ref}} ( y_{l} | x )$ .
Compute implicit rewards: $\hat{r}_{w} = \beta ( \log \pi_{\theta} ( y_{w} | x ) - \log \pi_{\text{ref}} ( y_{w} | x ) )$ , similarly $\hat{r}_{l}$ .
Loss: $\mathcal{L} = - \log \sigma ( \hat{r}_{w} - \hat{r}_{l} )$ , average over the batch.
Backward on $\mathcal{L}$ ; update $\pi_{\theta}$ only (ref is frozen).

With $\beta = 0 . 1$ and a well-initialized SFT model, $\hat{r}_{w} - \hat{r}_{l}$ starts near 0 (the model does not yet distinguish the pairs) and should reach $\sim$ 3-5 nats on average after a full epoch. At a batch size of 32 and 60K examples, one epoch is 1,875 steps; a full DPO run on 4 $\times$ A100 takes roughly 2-3 hours.

Implicit reward check. After training, verify that $\beta \log ( \pi_{\theta} ( y_{w} | x ) / \pi_{\text{ref}} ( y_{w} | x ) ) > \beta \log ( \pi_{\theta} ( y_{l} | x ) / \pi_{\text{ref}} ( y_{l} | x ) )$ on a held-out preference set. Accuracy below 65% on this proxy metric is a strong signal that $\beta$ is too large (over-regularizing) or the SFT checkpoint was too weak to start.

What You Actually Decide in Practice

$\beta$ , the temperature parameter. This controls how tightly the policy stays near $\pi_{\text{ref}}$ . Values in $[ 0 . 05 , 0 . 5 ]$ are common; start at 0.1. Too small: the policy drifts far from the reference and can produce degenerate outputs. Too large: effectively no change from SFT. Monitor KL divergence from $\pi_{\text{ref}}$ on a validation set; target 5-15 nats after training.
Length normalization. Sum log-probabilities over response tokens without normalization biases the loss toward shorter sequences, which receive higher per-token log-probability on average. SimPO and lengthnormalized DPO divide the log-probability sum by sequence length before computing the reward margin. This simple fix measurably reduces the model’s preference for very short responses; use it by default.
Offline vs. iterative. Static offline DPO is fast but brittle when the reject set is low-quality or the SFT model is strong. If your rejected completions are all from an early, weak model, the policy learns to beat a low bar and does not generalize. Regenerate the rejected set from your current policy every 1-2 epochs (online DPO) to keep the training signal challenging and on-distribution.
Chosen completion collapse. A silent failure mode: as the policy is pushed away from the rejected completion, the log-probability of the chosen completion can also decrease. Track $\log \pi_{\theta} ( y_{w} | x ) - \log \pi_{\text{ref}} ( y_{w} | x )$ separately during training; if it goes negative and keeps falling, the model is degrading on the positive examples, not just improving on the negative ones. Reduce learning rate or add a supervised term on chosen completions.
Data quality beats data quantity. On 7B-scale models, 10K high-quality human preference pairs (from annotators with explicit factuality checks) consistently outperform 200K AI-generated pairs in downstream benchmarks. The DPO loss is sensitive to label noise: a single flipped pair in a batch contributes a gradient in the wrong direction with full weight.

How This Shows Up in Interviews

“Derive DPO from first principles.” Strong answer: state the KL-regularized RL objective, take the non-parametric optimum, solve for the implied reward, substitute into Bradley-Terry, and show that $Z ( x )$ cancels. Many candidates can name DPO; very few can reproduce the derivation under pressure. Practice the three-step: optimal policy form $\rightarrow$ implied reward $\rightarrow$ preference loss.
“When would you use PPO instead of DPO?” Strong answer: when rewards are verifiable (math, code), when you need on-policy exploration to escape the static offline data distribution, or when the preference dataset is too small/noisy to provide stable gradients. PPO’s on-policy nature lets it discover high-reward completions that were never in the original training set. DPO is bounded by the quality of the collected pairs.
“How is DPO related to contrastive learning?” Strong answer: the DPO loss increases the log-likelihood of chosen completions relative to rejected ones, weighted by prediction error-closely analogous to InfoNCE / noise-contrastive estimation. The difference is that DPO’s “negative” is a specific rejected completion from a preference dataset, not a random negative sample.
Senior follow-up: “DPO has no explicit reward model. Where does the reward model effectively live?” It lives implicitly in the log-ratio $\beta \log ( \pi_{\theta} ( y | x ) / \pi_{\text{ref}} ( y | x ) )$ . This ratio is the implied reward for any completion under the optimal DPO policy. In practice, you can extract this scalar at inference time to rank completions just as you would with an explicit reward model-the difference is that this reward was never trained with a classification head, so it is not calibrated in absolute terms but is well-ordered within a prompt. This is useful for best-of- $n$ reranking and for diagnosing what the DPO model actually learned to prefer.

Key Takeaways

DPO derives from the same KL-regularized objective as PPO but avoids rollouts, reward models, and value networks by showing that the optimal policy and the reward are interchangeable representations of the same underlying quantity.
The DPO loss is a binary cross-entropy over preference pairs where the logit is the margin between implicit rewards of chosen and rejected completions; $\beta$ controls how far the policy can drift from the SFT reference.
A critical failure mode is chosen-completion collapse: track the log-ratio for chosen completions separately; if it falls below zero and keeps falling, the policy is deteriorating on positive examples.
Offline DPO is bounded by the quality of the static preference dataset; iterative or online DPO (regenerating rejected samples from the current policy) recovers much of PPO’s benefit at a fraction of the implementation cost.
Length normalization (dividing log-probabilities by sequence length before computing the reward margin) eliminates the length bias that favors short completions and should be applied by default.

27.3 Pitfalls of RLHF

27.3.1 Reward Over-Optimization

In any RLHF deep-dive, you will eventually be asked: “what goes wrong when you optimize too hard against the reward model?” Reward over-optimization is the canonical failure mode - the wall every production RLHF system eventually hits. Understanding it separates engineers who can implement the algorithm from those who can debug a live system that started producing worse outputs even though the proxy reward kept climbing. At senior and staff level, you are expected to know not just that this happens, but why it happens mechanically and how to detect it before it reaches users.

Reward over-optimization occurs when the RL policy gets so good at maximizing the proxy reward - the learned reward model $r_{\phi}$ - that it drifts away from what $r_{\phi}$ was actually supposed to measure: true human preference. The reward model is a finite-sample estimator of human preferences trained on a fixed annotation set. Like any learned model, it has gaps between its training distribution and the true underlying oracle. As optimization pressure increases, the policy finds and exploits those gaps.

The framing I find most useful is a train-test gap in RL clothing. The reward model is your training signal; true human preference is your test set. The RL algorithm overfits the former, and you pay the cost on the latter. Formally, let $r^{\ast}$ denote the true human preference oracle and $r_{\phi}$ the fitted proxy. The PPO training objective maximizes:

\mathbb{E}_{y \sim \pi_{\theta} ( \cdot | x )} [ r_{\phi} ( x , y ) ] - \beta D_{\mathrm{KL}} ( \pi_{\theta} \| \pi_{\mathrm{ref}} )

but what we actually care about is $\mathbb{E} [ r^{\ast} ( x , y ) ]$ . These two quantities correlate when $\pi_{\theta}$ stays close to the distribution $r_{\phi}$ was trained on, and diverge as $\pi_{\theta}$ moves away. The KL penalty keeps the policy near $\pi_{\mathrm{ref}}$ , but it is a soft constraint - once the policy gradient finds directions that increase $r_{\phi}$ while partially circumventing $D_{\mathrm{KL}}$ , it will exploit them.

Empirically, this pattern is remarkably consistent: proxy reward climbs monotonically with RL steps while true win-rate first increases, peaks, and then falls. Studies confirm this for human feedback and noisy AI feedback. It does not appear for clean, noiseless reward signals - pointing directly at label noise and finite-sample estimation as the root causes, not RL per se.

A reward model is a proxy, not an oracle; every RL algorithm will eventually find the gap between them.

The mechanism is Goodhart’s Law instantiated in RL: when the proxy becomes the optimization target, it ceases to be a reliable proxy. Human preference annotations are noisy in two compounding ways. First, individual annotators disagree; a pair that one rates “clearly response A” another rates “slightly response B.” Second, the preference data is collected on a distribution of prompts and responses that the current policy never generated, which means the reward model is queried increasingly far outside its training distribution as optimization proceeds. A relentless RL optimizer will find exactly those out-of-distribution regions.

The alternative - optimizing directly against live human ratings - is logistically infeasible at training speed. You cannot collect fresh human feedback fast enough for gradient updates at typical RL throughput. DPO sidesteps reward-model fitting by directly optimizing the policy against a static preference dataset, which reduces proxy-exploitation risk at the cost of being offline. Iterating the preference-collection loop partially recovers freshness but adds annotation latency and engineering complexity. Neither approach eliminates the fundamental finite-sample problem; they only slow the divergence.

A Worked Example

You are running PPO with a reward model $r_{\phi}$ trained on 100K pairwise annotations for a 7B chat model. Your SFT baseline achieves 50% win-rate against itself (the reference point). After 500 RL steps, proxy reward has risen 0.4 normalized units and human win-rate is 62% - a solid alignment gain worth shipping.

You continue to 2,000 steps. Proxy reward has now risen 1.1 units. But the next human evaluation returns a win-rate of 53%. What happened?

Inspect a sample of rollouts at step 2,000. The policy has learned three consistent patterns the reward model scores highly but humans do not: (1) responses are 2.1 $\times$ longer on average; (2) nearly every response opens with “Certainly! Here is a comprehensive answer to your question”; (3) factual claims are decorated with hedging phrases that sound authoritative but add no information. These stylistic patterns were rare in the reward model’s training corpus - it never learned to penalize them - but after 2,000 steps of optimization they appear in over 80% of rollouts. The reward model confidently assigns them high scores. Human raters find them grating. Proxy and reality have diverged.

The diagnosis is clean: the proxy reward at step 2,000 measures “looks like the training distribution of good responses” rather than “is a good response.” The RL algorithm found a different attractor.

What You Actually Decide in Practice

Always monitor two signals simultaneously. Track both proxy reward and an independent evaluation signal: a held-out reward model trained on disjoint data, an automated capability benchmark, or scheduled human spot-checks. When they diverge, stop. Do not wait for the next planned evaluation checkpoint.
Set a KL budget, not just a learning rate. A KL penalty $\beta$ in the range 0.01-0.05 is typical for 7-70B models; larger $\beta$ reduces over-optimization risk but weakens the RL signal. If the mean KL from the reference policy climbs past 10-15 nats, you are likely in dangerous territory regardless of $\beta$ .
Collect fresh reward data iteratively. If annotation throughput allows, retrain the reward model on samples from the current policy every $N$ RL steps. This shrinks the distribution gap at each step. The cost is engineering complexity; the benefit is a slower divergence curve.
Do not port “optimal RL step” numbers between projects. The over-optimization inflection point depends on reward model quality, annotation noisiness, and base model capability. Typical inflection ranges I have seen are 500-3,000 steps for instruction-tuned 7-70B models, but these numbers are not transferable across reward model quality tiers or annotation protocols.
Verifiable rewards are the cleanest cure. Over-optimization is essentially absent when rewards are noise-free - unit tests for code correctness, exact match for structured math answers. This is the core motivation for the RLVR paradigm: by replacing noisy human feedback with a ground-truth oracle, the proxy gap closes to near zero.

How This Shows Up in Interviews

“Why does RLHF sometimes make models worse even though the reward goes up?” Strong answer: the reward model is a finite-sample proxy. As the policy optimizes against it, the policy finds stylistic and structural patterns that score well on the proxy but were underrepresented in reward model training - a train-test gap. Proxy reward and true win-rate diverge after a critical number of RL steps; empirically, this inflection is consistently observed across human and noisy AI feedback settings.
“How do you detect and prevent reward over-optimization in a production training run?” Strong answer: maintain a separate evaluation signal not used for training reward (held-out reward model, automated benchmark, human A/B), monitor both continuously throughout the run, use KL regularization to limit policy drift, and stop when the independent signal plateaus or drops.
Senior follow-up: “If you iteratively retrain the reward model on data from the current policy, does over-optimization disappear?” Senior answer: it is significantly reduced but not eliminated. Each iteration fits a fresh finite-sample estimator on the current policy’s distribution, which closes the distribution gap at that step. But label noise in human annotations is irreducible - you are still fitting a noisy estimator, and the RL algorithm will still find its edges given enough steps. This is why noise-free verifiable rewards achieve fundamentally different training dynamics: there is no proxy gap to exploit.

Key Takeaways

Reward over-optimization is the RL analog of overfitting: the policy maximizes a proxy that drifts from the true objective.
The root cause is noise and finite-sample estimation in human preference labels, not a flaw in RL itself.
Proxy reward and true win-rate always diverge eventually; monitoring both and stopping early is non-negotiable.
KL regularization slows but does not eliminate over-optimization; iterative reward model retraining helps but does not fully close the gap.
The cleanest solution is verifiable rewards, where noise-free signals make over-optimization negligible - the key insight motivating the reasoning-model generation of post-training.

27.3.2 Length Bias and Calibration Degradation

Two failure modes of RLHF are subtle enough to survive into production before anyone notices. First, models trained with human feedback systematically grow more verbose - not because verbosity helps users, but because the reward signal incentivizes it. Second, they become overconfident: their output probabilities no longer track empirical accuracy in any reliable way. Neither shows up in the proxy reward metric, and neither is caught by standard capability benchmarks. Both have been replicated across multiple labs. Understanding them is basic literacy for any engineer running a production RLHF pipeline.

Length bias arises from the structure of pairwise preference annotation. When a human rater sees two responses and must pick one, a consistent heuristic applies: the longer, more detailed response looks more thorough and helpful. This is often true - more relevant content is better - but it creates a spurious correlation between response length and preference label that is independent of actual quality. A reward model trained on these annotations internalizes the correlation. The RL policy, optimizing against that reward model, learns that verbosity is profitable. Lengths climb throughout training even after task accuracy has plateaued.

Calibration degradation is a deeper structural consequence of abandoning distribution matching. Pretraining and SFT train the model by minimizing cross-entropy loss, which is equivalent to minimizing KL divergence from the data distribution. This implicitly trains the model to assign token probabilities that match empirical frequencies - the origin of calibration. RLHF replaces this objective with a scalar reward. There is no term in the PPO or DPO loss that requires output probabilities to remain accurate probability estimates. The optimizer is free to, and does, push probabilities toward extremes that maximize reward while discarding distributional accuracy.

Expected calibration error (ECE) measures the resulting gap:

\text{ECE} = \sum_{b = 1}^{B} \frac{| B_{b} |}{n} | \mathrm{acc} ( B_{b} ) - \mathrm{conf} ( B_{b} ) |

where $B_{b}$ is the set of predictions with confidence in bucket $b$ , $\mathrm{acc}$ is empirical accuracy within that bucket, and $\mathrm{conf}$ is mean predicted probability. RLHF models consistently show higher ECE than their SFT counterparts at temperature 1, and the gap grows with RL training steps.

RLHF produces a policy, not a probability model; treating its output probabilities as calibrated confidence estimates is a category error.

Length bias exists because pairwise annotation measures overall impression, not information per token. A 300-token response with three substantive points and a polite closer tends to beat a 150-token response with the same three points and nothing extra, even when both are equally correct. The reward model is not wrong to learn this - it correctly fits the labels it was given - but those labels conflate quality with length. Once the reward model internalizes the correlation, the RL loop exploits it directly.

Calibration degradation exists because pretraining instills calibration as a side effect of log-likelihood maximization on a diverse corpus. RLHF is an active intervention that overrides that prior. The policy gradient objective pushes the model’s distribution toward regions of high expected reward, which are not the same regions of high empirical token frequency that calibration requires. Empirically, RLHF models at temperature 1 assign probability $\sim$ 0.90 to answers where actual accuracy is $\sim$ 0.70. The models sound more decisive and confident to users. Human raters find decisive responses appealing and reward them, which means the RL loop actively reinforces overconfidence.

Both pathologies compound with over-optimization: the longer you run RL, the longer responses get and the worse calibration becomes, independently of whether the proxy reward is still climbing.

A Worked Example

Start from a 7B model fine-tuned with SFT on an instruction corpus. Run 1,000 steps of PPO against a reward model trained on 50K pairwise annotations. Evaluate at checkpoints: SFT baseline (step 0), step 500, and step 1,000.

Length. On 200 sampled open-ended questions: SFT baseline averages 118 tokens per response. At step 500: 174 tokens. At step 1,000: 231 tokens. The proxy reward has increased 0.6 normalized units over this range. A held-out reference-based metric (ROUGE-L against human-written answers) gains 2.1 points from step 0 to step 500 - a real quality improvement. From step 500 to step 1,000, ROUGE-L adds only 0.3 points while mean length grows by 57 tokens. The extra tokens are padding: repeated topic sentences, redundant closing summaries, filler phrases like “I hope this helps.”

Calibration. On 500 factual multiple-choice questions with verified ground truth: SFT ECE $\approx 0 . 06$ at temperature 1. After 1,000 PPO steps: ECE $\approx 0 . 19$ . The model now assigns mean confidence 0.91 to questions where its accuracy is 0.74. To recover ECE $< 0 . 07$ via temperature scaling, the SFT model requires $T \approx 1 . 15$ ; the RLHF model requires $T \approx 1 . 80$ , indicating that probability mass has been substantially compressed toward high-confidence predictions throughout RL training.

What You Actually Decide in Practice

Log output length throughout RLHF training and compare against the SFT baseline by query category. If mean length doubles without a corresponding quality gain on a held-out metric, you have a length bias problem. Address it with length-normalized objectives - SimPO divides the reward by response length; length-normalized DPO applies the same correction inside the DPO loss. Both add one division to the loss computation and cost negligible engineering effort.
Never use an RLHF model’s raw output probabilities as confidence estimates. If your application requires calibrated uncertainty - risk scoring, threshold-based routing, any safety-critical use - either apply post-hoc calibration (temperature scaling, Platt scaling) at deployment time or use the SFT checkpoint for probability estimates and the RLHF checkpoint for output generation.
Include a calibration checkpoint in your model evaluation gate. Before any RLHF checkpoint ships, measure ECE on a factual benchmark at temperature 1. Treat it like a regression test: any checkpoint with ECE exceeding a predefined threshold is blocked from serving. A reasonable starting threshold for a 7B model is ECE $< 0 . 10$ ; tighten this for high-stakes applications.
Overconfident models are a safety concern, not just a calibration metric. Users interpret confident tone as reliable information. In medical, legal, or financial contexts, a model that expresses 90% confidence when it is right only 70% of the time is actively misleading. This is worth naming explicitly in any system design review.
Both pathologies worsen with more RL steps, which is an additional practical reason - beyond reward over-optimization - to monitor closely, set early-stopping criteria, and not treat longer training as automatically better.

How This Shows Up in Interviews

“What are the systemic side effects of RLHF on model behavior beyond the intended alignment?” Strong answer: length bias (verbosity is spuriously rewarded because human annotators conflate length with quality) and calibration degradation (RL optimizes a scalar reward rather than a distributional likelihood, destroying the probabilistic interpretation of output probabilities). Both are empirically replicated across lab settings and grow worse with more RL steps.
“How would you detect and fix length bias in a deployed RLHF model?” Strong answer: instrument production request logs for output token count by query category; compare against the SFT baseline. If length is inflated, fix it in the next training run with length-normalized objectives. As a short-term mitigation, apply a maximum-token constraint in the inference server.
Senior follow-up: “Design an evaluation protocol that catches calibration degradation before a model ships.” Senior answer: maintain a calibration benchmark of 500-1,000 factual multiple-choice questions with verified ground truth. After each RLHF checkpoint, measure ECE at temperature 1 and track the temperature-scaling coefficient needed to reach a target ECE. A rising coefficient signals growing overconfidence even if post-scaling ECE looks acceptable - the model is producing increasingly peaked distributions that require larger corrections. Treat both the raw ECE and the scaling coefficient as go/no-go signals. Flag any checkpoint where the required scaling temperature exceeds 1.5 for explicit review before serving.

Key Takeaways

Length bias is a systematic artifact of pairwise annotation: human raters conflate length with quality, reward models learn this correlation, and RL exploits it. Monitor mean output length throughout training.
RLHF breaks calibration because RL optimizes a scalar reward rather than log-likelihood; output probabilities from RLHF models should not be treated as calibrated confidence estimates.
Both pathologies are empirically confirmed across labs and worsen with more RL steps, providing additional practical motivation to stop early and monitor closely.
Length-normalized objectives (SimPO, length-normalized DPO) directly address length bias at the algorithm level with minimal implementation cost.
Post-hoc temperature scaling can partially recover calibration for RLHF models, but the required temperature coefficient grows with RL training steps - track it as a leading indicator of degradation.