Chapter 28 Reinforcement Learning from Verifiable Rewards

28.1 From RLHF to RLVR

28.1.1 Why Verifiable Rewards Scale Better

When an interviewer asks you to design a post-training pipeline for a reasoning model-one capable of solving olympiad-level math or generating working code-you will eventually be asked why RLHF alone is not enough. The failure mode is overoptimization, and knowing its exact mechanism is what distinguishes a senior answer from a vague one. Reinforcement learning from verifiable rewards (RLVR) is the architectural shift powering every frontier reasoning model released in the past two years: o1, DeepSeek R1, Kimi K1.5, and Qwen 3 all rely on it. This topic appears in design-round and deep-dive interviews at virtually every lab working on post-training.

RLHF trains a policy to maximize a learned reward model fitted on human pairwise preferences. That reward model is a proxy: a finite neural network trained on a finite set of comparisons, approximating an underlying human preference distribution that is both noisy and complex. As gradient descent pushes the policy against this proxy, the policy learns to exploit the gaps between the proxy and the true preference function-a phenomenon called reward overoptimization. The proxy score climbs, but real human win rates plateau and eventually decline.

RLVR replaces the learned reward model with a deterministic, ground-truth verifier. In math, the verifier checks whether the model’s final answer matches the reference solution, either via regex or a symbolic equivalence library. In code, it executes the generated function against a hidden test suite and returns pass/fail. The reward is binary and exact. Because the verifier is not a neural network, it cannot be gamed by the gradient-there is no gap between the proxy and the ground truth, because they are the same object.

The formal RL objective is identical in both paradigms:

The entire difference lives in $r ( x , y )$. In RLHF, $r$ is a neural classifier with hundreds of millions of parameters, updated on human labels, and subject to distributional shift. In RLVR, $r$ is a deterministic function: a regex match, a SymPy symbolic check, or a unit-test runner. The verifier has zero learnable parameters, evaluates in microseconds, and cannot be overfit by gradient descent.

RLVR succeeds where RLHF fails not because the RL algorithm changes, but because the reward signal is ground truth rather than a learned approximation of it.

The practical ceiling of RLHF arises from two compounding problems. First, human pairwise preferences are expensive to collect-labeler programs at scale cost roughly $1-$5 per comparison after quality control, and a production RLHF run consumes millions of comparisons. This makes it structurally impossible to generate reward signal online during training at the rate a GPU cluster can consume it. The reward model is trained once, fixed, and then used as a static oracle, even as the policy drifts far from the distribution the reward model was trained on.

Second, human preferences are noisy. Individual annotators disagree, preference labels carry positional and length biases, and the reward model fits to these artifacts. Empirically, when researchers train RLHF reward models on noiseless synthetic preferences, overoptimization disappears almost entirely; it reappears immediately when noise is reintroduced. This tells you that overoptimization is not a failure of RL algorithms but a failure of the signal source.

RLVR bypasses both problems. A regex match costs a fraction of a cent per sample, so you can evaluate reward on every rollout during training with no budget pressure. The verifier is noiseless and stationary-a correct answer is always correct, regardless of how the policy evolved. This is the same design insight behind AlphaGo and AlphaFold: once you have a domain where correctness is computable, RL scales in proportion to compute, not in proportion to human annotation bandwidth.

RLHF also silently degrades calibration. Because the policy gradient optimizes expected reward rather than log-likelihood of any distribution, the resulting model has no probabilistic grounding. Multiple independent results-from Anthropic, from the GPT-4 technical report, from academic studies-show that RLHF models are substantially overconfident at temperature 1 relative to their SFT checkpoints. If you are building a system where calibrated probabilities matter (uncertainty-aware retrieval, best-of-$n$ selection, or chain-of-thought aggregation), this is a real operational liability. RLVR does not fix calibration, but it at least operates in a domain where calibration is less central to the deployment use case.

A Worked Example

Consider training a 7B base model (32 layers, $d = 4096$, bf16) with RLVR on a curated set of competition math problems using GRPO on 8 H100s. Group size is $G = 8$ completions per prompt.

For a prompt $x$ such as “Find all real solutions to $x^{2} - 5 x + 6 = 0$”, the policy samples 8 responses. Each is graded by a SymPy equivalence checker: responses stating $x = 2$ and $x = 3$ (in any equivalent form) receive $r_{i} = 1$; all others receive $r_{i} = 0$.

Suppose the group rewards for one prompt are $[ 1 , 1 , 0 , 1 , 0 , 0 , 1 , 1 ]$, giving a group mean $\bar{r} = 0 . 625$ and standard deviation $\sigma_{r} \approx 0 . 48$. The GRPO advantage for each response is:

Correct responses receive $A_{i} \approx + 0 . 78$; incorrect responses receive $A_{i} \approx - 1 . 30$. The policy gradient step increases the log-probability of tokens in correct responses and decreases it for incorrect ones-with no reward model, no value network, and no human in the loop.

Notice what happens at the extremes. If all 8 responses are correct ($\bar{r} = 1$, $\sigma_{r} = 0$), the advantages collapse to zero and the gradient vanishes-this problem is too easy and contributes nothing. If all 8 are wrong ($\bar{r} = 0$, $\sigma_{r} = 0$), the same collapse occurs-this problem is too hard. Only problems where the group pass rate sits between roughly 20% and 80% generate informative gradient signal. Filtering your training set to this difficulty band is one of the highest-leverage decisions in an RLVR pipeline.

After a few thousand gradient steps across an 8-H100 run, a 7B model trained this way typically improves 15-25 percentage points on MATH-500 relative to the SFT checkpoint. A comparable RLHF run on open-ended preferences would require careful early stopping, and the gains would be harder to reproduce across seeds.

What You Actually Decide in Practice

• Domain selection is the upstream constraint. RLVR only applies where you can write a verifier. Math (symbolic equivalence via SymPy), code (unit-test execution), and formal proofs (proof-checker APIs) qualify. Summarization, creative writing, and general instruction following do not. For those domains, RLHF or AI-judge feedback remains the only option.

• Format rewards are not optional. Every production RLVR system pairs the accuracy reward with a lightweight format reward-typically a small bonus for placing the chain of thought inside designated de-limiters such as <think>...</think>. Without format rewards, models produce structurally inconsistent outputs that are difficult to verify or display. In practice, the format reward is as operationally critical as the accuracy reward for training stability.

• Verifier quality is the new reward-model quality. Raw string matching on math answers fails on equivalent but syntactically different expressions (e.g., $\frac{1}{2}$ vs. $0 . 5$). Use symbolic equivalence (SymPy) or an LLM-as-judge for math, and hidden test cases with edge coverage for code. A leaky verifier creates exactly the same overoptimization dynamic as a noisy reward model.

• Problem difficulty filtering is a first-class concern. Sample only problems where your current policy’s group pass rate falls between 20% and 80%. Problems outside this range contribute near-zero gradient in expectation. Kimi K1.5 explicitly excludes problems solvable by best-of-8 from their base model; Qwen 3 achieves strong reasoning gains from only about 4,000 curated problems filtered to this difficulty window.

• RLHF still follows RLVR in production. RLVR sharpens domain competence; it does not produce a well-rounded assistant. DeepSeek R1, Kimi K1.5, and Qwen 3 all apply a standard RLHF stage after RLVR to recover helpfulness, formatting quality, and safety on open-ended tasks. The two stages are complementary, not competitive.

How This Shows Up in Interviews

• “Why does RLHF fail for reasoning tasks?” Strong answer: the reward model is a noisy, finite proxy for human preferences. The policy exploits the gap between proxy and ground truth-reward scores rise while real quality stagnates. This is Goodhart’s law in practice. Human feedback is also too expensive to generate online at training scale.

• “How would you design a reward function for a math RL training pipeline?” Strong answer: binary accuracy reward using symbolic equivalence (SymPy) rather than string matching, plus a lightweight format reward for response structure. Filter problems to a 20%-80% group pass-rate window. Do not use multiple-choice or true/false problems-the model can random-walk to high reward without reasoning.

• Senior follow-up: “If RLVR is strictly better for reasoning, why do production pipelines still run RLHF afterward?” Senior answer: RLVR is better only within its domain. Once math and code competence are locked in, the model needs to handle open-ended tasks-writing, instruction following, multi-turn chat-where no verifier exists. RLHF with AI-judge feedback is the only practical signal source for those domains. The two stages address orthogonal capabilities and are used sequentially in all known production reasoning-model pipelines.

Key Takeaways

• RLHF overoptimizes because the reward model is a noisy, learnable proxy; the policy exploits the proxy faster than human win rates improve.

• RLVR uses deterministic verifiers-regex, symbolic checkers, unit tests-eliminating the proxy and making overoptimization structurally impossible against the reward function.

• Format rewards are a critical implementation detail: without them, RLVR pipelines produce structurally inconsistent outputs and train unstably.

• Only problems with a 20%-80% group pass rate generate meaningful gradient signal; filtering to this difficulty window is one of the highest-leverage decisions in RLVR.

• RLVR and RLHF are complementary: production pipelines use RLVR to build domain competence, then RLHF to recover general-purpose alignment on open-ended tasks.

28.1.2 Outcome Rewards vs. Process Rewards

When you are designing the reward signal for a math or code RL pipeline, the most consequential decision is not which optimization algorithm to use-it is where in the response to measure correctness. Do you score only the final answer, or do you also score each intermediate reasoning step? This outcome-versus-process distinction drives data cost, training stability, and the ceiling on model quality. It shows up in system design interviews as a follow-on after the basic RLVR setup is established, and answering it precisely-with the empirical context from R1 and DeepSeekMath-is what marks a senior answer.

An outcome reward is a single scalar assigned to the entire response based solely on whether the final answer is correct. The reward function $r ( x , y )$ ignores all intermediate reasoning: chains of thought, intermediate calculations, self-correction steps. It asks exactly one question: did the model produce the right final answer?

A process reward (or process reward model, PRM) assigns a score to each intermediate reasoning step. Given a problem $x$ and a chain of thought $y = ( s_{1} , s_{2} , . . . , s_{T} , a )$ where $s_{t}$ are reasoning steps and $a$ is the final answer, a PRM assigns a scalar $r_{t} \in [ 0 , 1 ]$ to each step $t$. The total reward is typically the minimum step score, a product, or a running average. The key advantage is that the PRM provides gradient signal at every step in the chain, not only at the terminal token.

Outcome rewards are cheap to evaluate and structurally immune to overoptimization; process rewards provide denser gradient signal but require expensive annotation and introduce a learned proxy that can itself be exploited.

Outcome rewards dominate production systems for three concrete reasons.

First, they require no labeled data beyond the final answer, which in verifiable domains is already known deterministically. A SymPy equivalence check or a test-suite runner costs microseconds and involves no human. A PRM, by contrast, requires either human annotation of individual reasoning steps-which is roughly $10 \times$ more expensive per example than annotating final answers-or a separately trained neural scorer. That scorer is itself a learned proxy, which reintroduces the overoptimization risk that RLVR was designed to escape. A miscalibrated PRM can penalize creative but ultimately correct reasoning, or reward plausible-looking but incorrect chains of thought that happen to resemble training examples.

Second, PRMs are uniquely difficult to calibrate for long chains of thought. A step that appears wrong at position $t = 3$ might be a deliberate detour that arrives at the correct answer by $t = 15$. Human annotators disagree heavily on intermediate mathematical steps, and model-based PRMs trained on synthetic labels compound their errors multiplicatively across a long chain. The effective noise of a PRM on a 64-step chain of thought is substantially higher than the noise of a binary outcome verifier on the same problem.

Third, the large-scale empirical evidence has settled, for now, in favor of outcome rewards. DeepSeekMath showed process supervision outperforming outcome supervision for a mid-sized base model on competition math-the PRM line was the best-performing method in that comparison. But when DeepSeek R1 scaled to a much larger base model with careful difficulty filtering and GRPO’s group baseline, they tried extensive PRM experiments and concluded that outcome-based rewards matched or exceeded PRM performance. R1 also evaluated Monte Carlo Tree Search as an alternative dense-feedback mechanism and reached the same conclusion: neither PRMs nor search-based methods added consistent value over outcome rewards with GRPO. The result that made PRMs look compelling in DeepSeekMath was contingent on model scale and dataset composition; it did not transfer to the R1 setting.

The main practical failure mode of outcome rewards is sparse reward: when the model is too weak to solve any problems in a batch, every rollout receives reward zero, the GRPO group advantage collapses to zero everywhere, gradient updates vanish, and training stalls. This is a scenario where process rewards would provide non-zero signal on partially correct reasoning. The practical resolution, however, is difficulty filtering rather than switching to PRMs-keep only problems where the group pass rate falls between roughly 20% and 80%, which guarantees that at least some rollouts receive nonzero reward and the group advantage is informative.

A Worked Example

Consider the two-step problem: “If $3 x + 7 = 22$, what is $2 x$?” Ground truth: $x = 5$, so $2 x = 10$.

Under an outcome reward, the verifier reads only the final answer token. A response that correctly computes $3 x = 15$, then $x = 5$, but writes “therefore $2 x = 9$” due to an arithmetic slip receives reward 0. The correct intermediate steps contribute nothing.

Under a process reward, a step scorer reads each line. Step 1: “$3 x = 22 - 7 = 15$” scores 1.0. Step 2: “$x = 5$” scores 1.0. Step 3: “$2 x = 9$” scores 0.0. The minimum-step PRM gives reward 0.0, a product gives 0.0, and an average gives 0.67. All formulations agree the final step is wrong; they disagree on how much credit the correct preceding steps deserve.

Now run $G = 8$ rollouts. Suppose 3 correctly produce $2 x = 10$ (reward 1) and 5 do not (reward 0). Under outcome rewards, the group mean is 0.375 and the GRPO advantage for a correct response is positive: the gradient step increases the log-probability of correct completions. If instead all 8 rollouts get the final step wrong, but 6 of them show correct intermediate algebra, a well-calibrated PRM returns non-zero group advantages-training does not stall. This is the canonical scenario in which process rewards add value.

But building this PRM cost human annotators several hours of step-level annotation, or required training a separate neural scorer on hundreds of thousands of labeled derivations. For a 7B model that can already solve 30% of problems at outcome level after difficulty filtering, this cost is rarely justified.

What You Actually Decide in Practice

• Default to outcome rewards; switch to PRMs only when filtering cannot rescue sparse rewards. If best-of-$G$ on your filtered training set still yields near-zero pass rate for the current policy checkpoint, a PRM may be necessary. This scenario is unusual for base models above 7B parameters on standard competition math.

• Treat difficulty filtering as the first-pass fix for sparse outcome rewards. Before investing in a PRM, filter problems so that the group pass rate sits between 20% and 80%. This directly addresses the sparsity problem that motivates PRMs. Kimi K1.5 and Qwen 3 both use this filtering strategy and achieve strong results without process supervision.

• GRPO already has a process-like component through per-token KL regularization. In the full reward-shaping setup, the per-token reward for token $t$ is the negative KL penalty: $- \beta \log ( \pi_{\theta} ( a_{t} |$ $s_{t} ) / \pi_{\mathrm{ref}} ( a_{t} | s_{t} ) )$. The accuracy reward appears only at the terminal token. This means GRPO is already a hybrid: dense regularization signal at every position, sparse task reward only at the end. The per-token KL is not a process reward in the supervised sense, but it prevents the model from taking arbitrarily divergent intermediate paths.

• Remove length normalization when using outcome rewards. Dividing the reward by response length-a choice in some early GRPO implementations-creates a perverse incentive: when reward is negative (incorrect response), the loss is minimized by making the response as long as possible so each token’s contribution is small. This produces models that pad incorrect responses to extreme lengths. Control response length through a separate, decoupled length reward rather than by normalizing the accuracy reward.

• If you do build a PRM, use a model-based equivalence checker rather than human step labels. Math shepherd-style approaches train a PRM on process annotations derived from large-scale rollouts: generate many responses, identify which steps appear in correct versus incorrect completions, and use that signal to train the scorer. This amortizes annotation cost but introduces its own distribution shift and noise. Budget for a calibration evaluation before deploying the PRM in a training loop.

How This Shows Up in Interviews

• “What’s the difference between outcome and process rewards, and when would you use each?” Strong answer: outcome rewards score only the final answer-deterministic, costless, no learned scorer. Process rewards score intermediate steps-denser gradient signal, but require expensive step-level annotation or a learned PRM that introduces proxy noise. Default to outcome rewards; consider PRMs only when difficulty-filtered outcome rewards still yield near-zero group pass rates across your training set.

• “Your RLVR training curve is flat and reward isn’t improving. What do you diagnose first?” Strong answer: check the group pass-rate distribution across your training problems. If the majority of problems have 0% or 100% group pass rate, you have a difficulty calibration problem, not an algorithm problem. Re-filter the problem set before adding a PRM or changing the optimizer. Also verify that your verifier is not silently accepting incorrect formats.

• Senior follow-up: “DeepSeekMath showed PRMs clearly outperforming outcome rewards. DeepSeek R1 used outcome rewards and dropped PRMs after extensive experiments. How do you reconcile those results?” Senior answer: DeepSeekMath used a smaller base model on hard competition problems, so outcome rewards were genuinely sparse-the model solved too few problems to receive useful signal. At R1 scale (a large pre-trained model with strong mathematical priors), enough problems in the filtered set receive nonzero outcome reward for GRPO’s group baseline to work well. The PRM advantage is a function of model weakness and problem difficulty relative to the current checkpoint; as those improve, the marginal value of a PRM decreases while its engineering cost remains constant. R1’s conclusion is conditional on their filtering and model scale, not a universal result.

Key Takeaways

• Outcome rewards assign a single binary score to the final answer; process rewards (PRMs) assign scores to each intermediate reasoning step, providing denser gradient signal at higher annotation cost.

• Outcome rewards win in production systems because they are deterministic, require no learned scorer, and avoid the secondary overoptimization risk a PRM introduces.

• Sparse outcome rewards are best addressed by difficulty filtering (target 20%-80% group pass rate), not by adding a PRM.

• GRPO is already a hybrid: it applies per-token KL regularization at every position (dense regularization) with the accuracy reward concentrated at the terminal token (sparse task signal).

• The PRM advantage observed in smaller-scale systems diminishes as base model capability improves and difficulty filtering is applied; PRMs remain a viable tool for bootstrapping RL on weak policies, not a permanent fixture of mature pipelines.

28.2 Policy Gradient Methods

28.2.1 Baselines, Advantages, and Variance Reduction

At a senior post-training interview, after you sketch out any RL training loop-PPO, GRPO, or a custom variant-the follow-up is almost always: “Walk me through why your gradients are noisy and what you do about it.” Variance in policy gradient estimates is the central numerical challenge that separates working implementations from broken ones. Engineers at L5-L6 are expected to derive the baseline trick on a whiteboard, prove it is bias-free, and explain how it connects to the advantage function that every practical algorithm uses internally.

The policy gradient theorem gives the gradient of expected reward $J ( \theta ) = \mathbb{E}_{a \sim \pi_{\theta} ( \cdot | s )} [ R ( s , a ) ]$ as:

The derivation uses one identity: $\nabla_{\theta} \pi_{\theta} = \pi_{\theta} \nabla \log_{\theta} \pi_{\theta}$. In the language model setting, $s$ is the prompt, $a$ is the complete response, and $R$ is a scalar reward-binary for verifiable tasks (correct/incorrect). Estimating this gradient from a single rollout is unbiased but has enormous variance. If a model generates math proofs and achieves a 5% solve rate, 95% of gradient samples are exactly zero and training makes no progress.

The baseline trick is the structural fix. Any function $B ( s )$ that depends only on the state (not the action) can be subtracted from the reward without biasing the gradient:

The proof is one line: $\mathbb{E}_{a \sim \pi_{\theta}} [ B ( s ) \nabla \log_{\theta} \pi_{\theta} ( a | s ) ] = B ( s ) \cdot \mathbb{E}_{a} [ \nabla \log_{\theta} \pi_{\theta} ( a | s ) ] = 0$, because the gradient of a normalized log-probability sums to zero over any action space.

The optimal baseline (minimizing gradient variance) is $B^{\star} ( s ) = \mathbb{E}_{a \sim \pi_{\theta}} [ R ( s , a ) | s ]$, the expected reward from state $s$. Practitioners call this the value function $V ( s )$. The difference between actual reward and this baseline is the advantage function:

A positive advantage says action $a$ outperformed the policy’s average expected outcome from $s$; a negative advantage says it underperformed. All practical policy gradient algorithms-PPO, GRPO, and every variant in between-are ultimately different answers to one question: how do we estimate $A ( s , a )$ cheaply and with low variance?

Every working RL algorithm for language models boils down to: compute an advantage estimate, upweight high-advantage responses, downweight low-advantage ones, and prevent the policy from moving too far in one step.

Without a baseline, raw reward conflates two signals: how good the response was and how easy the prompt was. A hard problem scoring 0.6 and an easy problem scoring 0.6 receive identical gradient weights, even though the former represents an excellent response and the latter a mediocre one. The baseline removes the prompt-level expected reward, isolating only the within-prompt quality signal.

This matters because sparse rewards make the naive estimator nearly useless even at large batch sizes. In supervised learning, every sample contributes a non-zero gradient. In sparse-reward RL, a 0-reward rollout contributes exactly zero-scaling the batch just scales the waste. Centering by the per-prompt mean turns every wrong answer into an active downweight signal, converting a sparse signal into a dense training signal across all rollouts in the group.

The value function approximation also adapts to question difficulty automatically. A baseline estimated from group rollouts is high for easy prompts (where most responses succeed) and low for hard prompts (where most fail). The resulting advantage correctly flags when a response is surprisingly good for its difficulty level, rather than just absolutely high-reward.

A Worked Example

Setup: 32 prompts, $G = 4$ rollouts each, from a 7B model on a math benchmark. Focus on two representative prompts:

• Prompt P1 (easy): rollout rewards $= [ 1 , 1 , 0 , 1 ]$. Mean $= 0 . 75$, $\text{std} \approx 0 . 43$.

• Prompt P2 (hard): rollout rewards $= [ 0 , 0 , 0 , 0 ]$. Mean $= 0$, $\text{std} = 0$.

Naive policy gradient: weight each rollout’s log-probability gradient by raw reward. P1-correct rollouts get weight $+ 1 . 0$; P1-wrong and all P2 rollouts get weight $0$. P2 contributes nothing to training even though the model is failing on it.

Centered advantages (subtract per-prompt mean):

Now P1’s wrong rollout receives a downweight of $- 0 . 75$, reinforcing the contrast between correct and incorrect responses. P2 correctly contributes zero gradient-there is no within-group signal when every rollout fails, and making a noisy update would only harm training.

As a concrete variance illustration: suppose two prompts contribute rewards of 11 and 9 (easy) or 0 and 2 (hard) under a uniform initial policy. Naive gradient variance across the four (state, action) pairs is Var$( \{ 11 , 9 , 0 , 2 \} ) = 21 . 25$. With baselines $B ($easy$) = 10$ and $B ($hard$) = 1$, effective rewards become $\{ + 1 , - 1 , - 1 , + 1 \}$ with variance $1 . 0$-a roughly 21$\times$ reduction at zero cost to the optimization objective.

What You Actually Decide in Practice

• Use $G \geq 4$ rollouts per prompt. With fewer samples, the empirical mean is a poor baseline estimate and may increase variance rather than reduce it. The GRPO paper uses $G = 8$; in practice $G = 4$-$8$ balances rollout inference cost against baseline quality.

• When all $G$ rollouts for a prompt achieve the same reward, centered advantages are zero and no update fires. This is correct behavior, not a bug. If every rollout is correct (easy) or every rollout is wrong (unsolvable), there is no within-group learning signal. Forcing an update from this zero-signal state only adds noise.

• Dividing centered advantages by the group standard deviation is not a valid baseline operation. Subtracting the group mean is bias-free (the subtracted quantity has expectation zero over actions). Dividing by $\sigma$ is not-it changes gradient magnitude as a function of reward spread. When $\sigma$ is small, gradients are amplified for prompts that are trivially easy or completely unsolvable, precisely the prompts where the gradient signal is least informative.

• A frozen initial checkpoint is a valid and cheap baseline. Treat $B ( s ) = \mathbb{E}_{a \sim \pi_{0}} [ R ( s , a ) | s ]$, estimated from a handful of rollouts under the pre-RL model, as a constant. No gradient flows through it. This approximates the value function without any learned network.

• Track per-prompt reward mean as a curriculum signal. Once the running success rate for a category of prompts exceeds 90%, those prompts contribute near-zero gradient signal at full inference cost. Filter them out and replace with harder examples.

How This Shows Up in Interviews

• “Why does vanilla policy gradient have high variance, and what is the baseline trick?” Name sparse rewards as the root cause. Derive $\nabla_{\theta} J = \mathbb{E} [ R \nabla \log_{\theta} \pi_{\theta} ]$ in one line via the log-derivative trick, then show subtracting $B ( s )$ is bias-free because $\mathbb{E}_{a \sim \pi_{\theta}} [ \nabla \log_{\theta} \pi_{\theta} ( a | s ) ] = 0$. The key constraint: $B$ must depend only on state, not action.

• “What is an advantage function and how does it connect to value functions?” The advantage $A ( s , a ) = R ( s , a ) - V ( s )$ is the signed difference between actual reward and the policy’s average expected reward from that state. In outcome-reward LM-RL, $V ( s )$ is approximated by the mean reward across $G$ rollouts for the same prompt-no future-reward bootstrapping is needed.

• Senior follow-up: “GRPO divides the group advantage by the standard deviation. Is that mathematically valid?” No. Subtracting the group mean is a valid baseline (zero-expectation subtraction over actions). Dividing by $\sigma$ is not-it modifies gradient magnitude as a function of reward spread, which has no justification in the policy gradient theorem. Specifically, when $\sigma \rightarrow 0$ (all rollouts score identically), gradients explode. The practical consequence is that problems at both extremes of the difficulty spectrumalready mastered or completely unsolvable-get disproportionate gradient weight. The “Dr. GRPO” analysis demonstrates that removing the $\sigma$ normalization yields shorter responses and equal-or-better reward, confirming this is a design flaw rather than a feature.

Key Takeaways

• The policy gradient $\nabla_{\theta} J = \mathbb{E} [ R ( s , a ) \nabla \log_{\theta} \pi_{\theta} ( a | s ) ]$ is unbiased but catastrophically high-variance under sparse rewards.

• Any state-dependent function $B ( s )$ is a valid bias-free baseline; the optimal choice is the value function $V ( s ) = \mathbb{E} [ R | s ]$.

• The advantage $A ( s , a ) = R ( s , a ) - V ( s )$ converts sparse rewards into dense relative signals across the rollout group, involving every rollout in training rather than only the successes.

• When all rollouts for a prompt tie on reward, centered advantage is zero and no update fires-this is correct behavior indicating no within-group learning signal.

• Dividing by group standard deviation (the GRPO default) is not a valid baseline operation; it overweights the difficulty extremes and can be safely removed without hurting performance.

28.2.2 GRPO in Depth

When you sketch an RL training loop for a 70B reasoning model and the interviewer asks, “PPO requires a critic as large as the policy-how do you handle that?” the answer is GRPO. Group Relative Policy Optimization removes the value network entirely, replacing it with a within-group reward normalization that fits naturally into the language model setting. Understanding the full algorithm-objective, training loop, memory layout, and failure modes-is now a baseline expectation for post-training roles at frontier labs, typically probed at L5 and above.

GRPO starts from PPO and makes one structural change: replace generalized advantage estimation (GAE), which requires a separately trained value network, with a group-level normalized reward. For each prompt $q$, sample $G$ responses $\{ a_{1} , . . . , a_{G} \}$ from the current policy $\pi_{\theta_{\text{old}}}$. The group advantage for response $i$ is:

where $\epsilon = 10^{- 4}$ prevents division by zero when all rollouts score identically. The GRPO objective applies the standard PPO clipped surrogate with this advantage:

where $r_{i} = \pi_{\theta} ( a_{i} | q ) / \pi_{\theta_{\text{old}}} ( a_{i} | q )$ is the importance ratio and $\varepsilon = 0 . 2$ is the clip radius. The KL term uses a control-variate estimator with lower variance than the standard Monte Carlo form:

This has the same expectation as $- \log ( \pi_{\text{ref}} / \pi_{\theta} )$ because $\mathbb{E}_{a \sim \pi_{\theta}} [ \pi_{\text{ref}} ( a ) / \pi_{\theta} ( a ) ] = 1$, so the extra term cancels in expectation while reducing variance of individual sample estimates.

In the purely online case where $K = 1$ gradient step is taken per rollout batch, the importance ratio $r_{i} = 1$ exactly, the clipping is inactive, and the loss reduces to a plain policy gradient weighted by $A_{i}$. The clipping and multi-step inner loop only engage when reusing rollouts across $K > 1$ gradient updates.

GRPO is PPO with the value network deleted and the advantage replaced by within-group z-score normalization; clipping, KL regularization, and the multi-step inner loop transfer directly from PPO unchanged.

The value network in PPO is typically the same architecture as the policy. A 7B policy in bf16 occupies 14 GB for parameters; with Adam optimizer states the footprint grows to roughly 56 GB. Adding a co-located 7B critic doubles the parameter memory to 112 GB before activations. On an 8$\times$H100 node with 640 GB total GPU memory, this leaves little headroom for activation memory and rollout buffers. Eliminating the critic is not a minor optimization-it is often the difference between a training run fitting on one node versus requiring two.

The group structure provides a difficulty-adaptive baseline without any learned parameters. A prompt where all $G$ rollouts score 1.0 produces $A_{i} = 0$ for every $i$: the policy has mastered it and no gradient flows. A prompt where 1 of 8 rollouts is correct produces $A_{\text{correct}} \approx + 1 . 63$ (the z-score of the single success), concentrating gradient mass precisely on the response that worked. This adaptive difficulty weighting is implicit and costs only a mean and standard deviation computation.

Compared to DPO, GRPO is inherently online: rollouts are sampled from the current policy at each step, rewards are computed immediately by a deterministic verifier (regex match, unit test, symbolic solver), and updates follow. DPO requires a static pre-collected preference dataset with an explicit pairing structure. When your reward is a fast deterministic checker-as it is for math, code, and logic tasks-on-policy GRPO can leverage the full capacity of the current model’s distribution while DPO is locked to the distribution of the fixed dataset.

A Worked Example

Setup: 7B model on a math benchmark, $G = 8$ rollouts per prompt, 32 prompts per batch, H100 80 GB, bf16, $\text{clip} \varepsilon = 0 . 2$, KL coefficient $\beta = 0 . 04$, $K = 4$ inner gradient steps.

Step 1 - Rollouts. For each of the 32 prompts, generate 8 responses via the inference engine (256 sequences total). Compute binary rewards from an exact-match checker. For one representative prompt, suppose rewards are $[ 1 , 0 , 0 , 1 , 0 , 0 , 1 , 0 ]$ (3 correct of 8).

Step 2 - Advantage computation.

Step 3 - Save old log-probs. At rollout time, compute and store $\log \pi_{\theta_{\text{old}}} ( a_{i} | q )$ for all 256 responses. This is one forward pass per response in inference mode. Store the result as a float32 array (256 sequences $\times$ 512 tokens $\approx$ 0.5 GB), not a copy of the model weights.

Step 4 - Clipped surrogate loss ($K = 4$ inner steps). For each inner step, compute $\log \pi_{\theta} ( a_{i} | q )$ under the current trainable policy and form $r_{i} = \exp ( \log \pi_{\theta} - \log \pi_{\theta_{\text{old}}} )$:

ratio   = (log_pi_theta - log_pi_old).exp()       # stop_grad on log_pi_old
surr1   = ratio * advantage                        # [G, L]
surr2   = ratio.clamp(1 - eps, 1 + eps) * advantage
pg_loss = -torch.min(surr1, surr2).mean()

Step 5 - KL penalty. Compute $\log \pi_{\text{ref}} ( a_{i} | q )$ from the frozen reference model and add:

kl_est  = (pi_ref / pi_theta) - (log_pi_ref - log_pi_theta) - 1
kl_loss = beta * kl_est.mean()
loss    = pg_loss + kl_loss

Step 6 - Gradient update. Standard Adam step. Repeat steps 4-6 for $K = 4$ iterations, then return to Step 1 for a fresh rollout batch.

Memory accounting: Policy parameters (bf16, 7B) $= 14$ GB; Adam states (fp32, 2$\times$7B) $= 56$ GB; $\pi_{\text{ref}}$ (frozen, bf16) $= 14$ GB; $\log \pi_{\text{old}}$ array $\approx 0 . 5$ GB. Total $\approx 85$ GB. No value network. Compare this to a PPO setup at the same scale: adding a 7B critic with its own Adam states would add another 70 GB, pushing the total past 150 GB.

What You Actually Decide in Practice

• Use $K = 1$ inner steps early in training. At $K = 1$, the importance ratio is exactly 1.0, clipping is inactive, and the loss is a plain policy gradient-the lowest-variance regime. Increase to $K = 4$-$8$ once the policy is stable enough that log-probs from one rollout batch remain reasonable approximations across multiple gradient steps.

• Never store $\pi_{\text{old}}$ as a full model copy. The only data you need from $\pi_{\text{old}}$ during the $K$ inner steps are the log-probs computed at rollout time. Storing 256 sequences $\times$ 512 tokens of log-probs costs $\sim$0.5 GB versus 14 GB for a second 7B model copy. Wrap the saved log-probs in torch.no_grad() to prevent accidental gradient flow.

• Watch response length as the primary pathology signal. When the GRPO loss divides per-token log-probs by sequence length, the per-token loss on a wrong response is $- | A | / L$. The model learns to minimize this by making wrong responses very long. If average response length increases monotonically while reward plateaus, remove length normalization from the loss immediately.

• Filter prompts where best-of-$G$ pass rate is 0% or 100%. At 0% pass rate, $\sigma_{R} = 0$ and advantages collapse or blow up. At 100%, the same collapse occurs and the prompt carries no gradient signal regardless of normalization. Excluding both extremes keeps every batch prompt informationally dense and implicitly implements a curriculum. Qwen 3 samples prompts proportional to $( 1 -$ pass_rate$)$ to enforce this dynamically.

• Use $\beta = 0 . 01$-$0 . 1$ for the KL coefficient. Too small and the policy drifts into reward-hacking regimes. Too large and the KL term dominates, starving reward optimization. Starting at $\beta = 0 . 04$ and monitoring KL divergence for the first few hundred steps is my preferred approach; if KL exceeds roughly 0.3 nats, increase $\beta$ or reduce the learning rate.

How This Shows Up in Interviews

• “How does GRPO differ from PPO, and when would you choose one over the other?” The core difference is the advantage estimator: GRPO uses within-group z-score, eliminating the value network and roughly halving memory. Choose GRPO when memory is the binding constraint or when you lack a good value function initialization (common in new task domains). Stay with PPO when you have a mature critic and need stronger variance reduction on long-horizon tasks where the group-mean baseline is insufficiently accurate.

• “Walk me through the GRPO training loop.” A complete answer covers: (1) rollout phase-sample $G$ responses per prompt, compute rewards and $\log \pi_{\text{old}}$ from a single forward pass; (2) advantage phasenormalize within group by subtracting mean and dividing by std plus $\epsilon$; (3) inner loop-compute importance ratios with a stop-gradient on $\log \pi_{\text{old}}$, apply clipped surrogate loss, optionally add KL penalty against $\pi_{\text{ref}}$, take gradient step; (4) repeat inner loop $K$ times, then refresh rollouts.

• Senior follow-up: “What happens to GRPO training when the model solves nearly all prompts in the training set?” This is the curriculum collapse scenario. As best-of-$G$ pass rate approaches 100% across most prompts, group standard deviation approaches zero everywhere. Advantages collapse (or explode with $1 / \sigma$ normalization) and training stalls despite the model still having room to improve on unseen distributions. The correct response is dynamic difficulty filtering: continuously measure pass rates, remove prompts above roughly 90% success, and inject harder prompts. Without this, GRPO training terminates not because the model is optimal but because the fixed training set has been exhausted.

Key Takeaways

• GRPO eliminates the PPO value network by replacing GAE with within-group reward normalization:

$A_{i} = ( R_{i} - R ) / ( \sigma_{R} + \epsilon )$, saving roughly 100% of critic parameter and optimizer-state memory.

• The PPO-clip objective and KL regularization are otherwise identical to PPO; the only structural change is the advantage estimator.

• The control-variate KL estimator $( \pi_{\text{ref}} / \pi_{\theta} ) - \log ( \pi_{\text{ref}} / \pi_{\theta} ) - 1$ equals $D_{\text{KL}} ( \pi_{\theta} \| \pi_{\text{ref}} )$ in expectation with lower per-sample variance than the standard Monte Carlo form.

• Store $\pi_{\text{old}}$ as a log-probability array (not a full model copy) to avoid a second model’s worth of memory during the inner gradient loop.

• Monotonically increasing response length signals the length-normalization incentive pathology; curriculum collapse when group $\sigma_{R} \rightarrow 0$ is the end-of-training failure mode-both require active monitoring and intervention.

28.3 Case Studies in Reasoning Models

28.3.1 DeepSeek-R1 and R1-Zero

When an interviewer asks you to design a reasoning model from scratch, they are really probing whether you understand the interaction between supervised fine-tuning and reinforcement learning with verifiable rewards. DeepSeek R1 and its controlled sibling R1-Zero are the clearest public demonstrations of this pipeline working end-to-end: released around the same time as OpenAI o1, they matched its performance on competitive math benchmarks using a recipe that is, in retrospect, surprisingly simple. This topic appears in senior-level design interviews at frontier labs because the engineering decisions-reward design, training stability, the role of coldstart data-reveal whether you can reason carefully about what each component actually contributes.

R1-Zero is the controlled experiment. Take DeepSeek V3 (the pretrained and mid-trained base model, before any instruction tuning or RLHF), and run pure RL using GRPO with verifiable rewards. No chain-ofthought (CoT) fine-tuning beforehand. No search or process supervision. Two rewards: a binary accuracy reward ($r \in \{ 0 , 1 \}$ for correct or incorrect answers on math and coding tasks) and a format reward that incentivizes the model to place its reasoning inside <think> $. . .$ </think> tags. The format reward feels like a cosmetic detail but turns out to be structurally important: without it, long CoTs do not emerge reliably from RL.

R1 is the production recipe. The pipeline has three stages:

1. SFT on long CoT data to cold-start the reasoning format.

2. RL with GRPO, using a binary accuracy reward, the format reward, and an added language consistency reward.

3. Standard post-training: instruction SFT combining reasoning and non-reasoning data, followed by RLHF using GRPO.

The GRPO advantage for response $i$ within a group $G$ of rollouts on the same prompt is:

The policy is then updated via the clipped surrogate objective inherited from PPO:

where $\rho_{i} = \pi_{\theta} ( o_{i} | q ) / \pi_{\theta_{\mathrm{old}}} ( o_{i} | q )$ is the likelihood ratio and $\epsilon = 0 . 2$ is the clip range.

The minimal viable RLVR recipe is a strong base model, binary correct/incorrect rewards on verifiable tasks, a format reward to stabilize CoT structure, and GRPO with group-normalized advantages.

Verifiable rewards eliminate the feedback loop that causes reward overoptimization in RLHF. When the reward is “is the math answer correct?”, it cannot be hacked: either the symbolic expression matches the reference, or it does not. There is no proxy reward model to overfit, and the signal does not degrade as you optimize. This is the core reason RLVR scales where RLHF stalls.

The SFT cold-start in R1 serves a specific purpose: it gives RL a reasonable starting distribution to explore from. Without it (R1-Zero), long CoTs do emerge eventually, but they can be incoherent and language-mixed-the model will switch mid-reasoning from English to Chinese and back, because RL has no incentive to stay in a single language. The language consistency reward in R1 provides that incentive explicitly. Some practitioners have found that even 1,000 long-CoT examples for SFT cold-start are enough to stabilize the format before RL.

Two clear negative results from the R1 paper deserve equal weight. Process reward models (PRMs), which provide intermediate per-step feedback on reasoning chains, did not outperform simple outcome-based rewards despite earlier promising results in the DeepSeekMath work. Monte Carlo Tree Search did not help either. The simpler, cheaper approach won: GRPO with outcome-only rewards, no tree search, no step-level annotation.

A Worked Example

Walk through the R1 training run with realistic numbers.

1. Base model: DeepSeek V3, a 671B Mixture-of-Experts model with 37B active parameters per forward pass.

2. SFT stage: Fine-tune on a set of long CoT trajectories (source not disclosed in the paper) to prime the model for structured reasoning.

3. RL stage: GRPO with $G = 8$ rollouts per prompt. Accuracy reward is binary: $r \in \{ 0 , 1 \}$. Clip ratio $\epsilon = 0 . 2$, KL coefficient $\beta = 0 . 001$.

4. Advantage computation example: Suppose a math problem yields 8 rollouts, 5 correct and 3 incorrect. Mean reward $\bar{r} = 5 / 8 = 0 . 625$, standard deviation $\approx 0 . 484$.

• Correct rollout: $A = ( 1 - 0 . 625 ) / 0 . 484 \approx + 0 . 77$. Policy gradient increases probability of this response.

• Incorrect rollout: $A = ( 0 - 0 . 625 ) / 0 . 484 \approx - 1 . 29$. Policy gradient decreases probability. For a trivially easy problem (all 8 correct), $\bar{r} = 1$ and $\operatorname{std} \rightarrow 0$, making $A$ numerically large. This is the bias introduced by standard deviation normalization, which the Dr. GRPO analysis identifies as distorting training on near-uniform reward groups.

5. Results: DeepSeek R1 matches OpenAI o1 across most benchmarks. On AIME 2024 it reaches approximately 79% pass@1; on MATH-500 it is near ceiling at 97%.

6. Distillation: Taking roughly 800,000 long-CoT traces from R1 and fine-tuning Qwen 32B via standard SFT on those traces lifts the student from $\sim$50% to $\sim$72% on AIME 2024. This is pure supervised distillation; no RL is needed for the student model.

What You Actually Decide in Practice

• Always do SFT cold-start before RL, even if it is small. R1-Zero is a research artifact; it languagemixes and ships poorly. Even 1,000 long-CoT examples stabilize the format. Budget data collection time before touching RL.

• The format reward is load-bearing. It seems cosmetic but multiple practitioners report that without rewarding the model for placing reasoning inside structured tags, GRPO does not reliably produce long CoTs. Add it from the start; do not treat it as optional.

• Add a language consistency reward. Aggressive RL on multilingual base models causes language mixing in CoTs. This is an optimization pressure artifact, not a base model defect. An explicit reward signal to stay in the user’s language is necessary for production.

• Do not invest in PRMs or MCTS until outcome rewards plateau. R1 tried both extensively and found neither improved over simple GRPO with outcome rewards. PRMs require expensive step-level annotation; MCTS adds infrastructure complexity. Optimize the simple thing first.

• SFT quality dominates RL algorithm choice. Empirical work (AI2’s Tulu 3) showed that improving the SFT stage closes most of the gap between DPO and PPO. If your reasoning RL is underperforming, check the SFT data before switching RL algorithms.

How This Shows Up in Interviews

• “Walk me through the training pipeline for a model like DeepSeek R1.” Strong answer: (1) start with a strong pretrained base model; (2) SFT on long CoT data to cold-start reasoning format; (3) GRPO RL with binary accuracy reward, format reward, and language consistency reward on verifiable tasks; (4) post-training SFT combining reasoning and general data; (5) RLHF using GRPO for instruction following. Make explicit that the RL stage in (3) is distinct from the RLHF stage in (5).

• “What reward signals would you use to train a reasoning model, and why not process rewards?” Strong answer: binary accuracy reward (correct/incorrect), format reward (structured tags), language consistency reward. PRMs require step-level human or model annotation and tend to be gamed; outcome rewards are fully self-supervised and, in the R1 experiments, sufficient.

• Senior follow-up: “R1-Zero shows CoT length growing during training and backtracking emerging. Is this evidence of genuine reasoning emergence?” Senior answer: treat this claim skeptically. The length growth is at least partially explained by the biased GRPO objective-standard deviation normalization upweights near-zero-variance problems (all correct or all incorrect), and the length normalization that vanilla GRPO applies creates perverse incentives to produce longer responses when the answer is wrong. Backtracking phrases like “wait, let me reconsider” appear in the base model at temperature above zero as well. R1-Zero produces a strong math model; attributing the CoT behavior to emergent reasoning rather than optimization artifacts requires stronger evidence than the paper provides.

Key Takeaways

• R1-Zero demonstrates that pure GRPO RL on verifiable rewards is sufficient for strong math reasoning; R1 adds SFT cold-start and language consistency rewards for production stability.

• The minimal reward design is binary accuracy plus format reward; language consistency reward prevents CoT language mixing under aggressive RL.

• Process reward models and Monte Carlo Tree Search both failed to outperform simple outcome-based GRPO in the R1 experiments.

• Distillation via SFT on $\sim$800,000 R1 CoT traces transfers most of the reasoning gain to a smaller student model without any RL.

• SFT data quality dominates RL algorithm choice: fixing the supervised stage routinely closes gaps that switching between RL algorithms cannot.

28.3.2 Kimi K1.5 and Qwen3

Kimi K1.5 and Qwen 3 arrived within weeks of DeepSeek R1, both independently matching OpenAI o1 on competitive benchmarks. Studying them alongside R1 is instructive precisely because they solve the same problem with different algorithmic choices and different engineering emphases, revealing which design decisions are load-bearing and which are incidental. In a system design interview, after you have described the R1 recipe, the follow-up is often: “Is there another way to do this?” or “How would you control the inference cost of a reasoning model?” These two papers are your answer.

Kimi K1.5 derives its RL objective from the same KL-regularized reward maximization problem as DPO, but without pairwise preferences. The Moonshot team starts from the optimality condition for the constrained policy:

Since $Z ( q )$ is a normalizing constant that cannot be computed in practice, they treat the right-hand side as a target and minimize a squared loss between the observed reward and the model-implied log ratio. Taking the gradient of this squared loss yields an update rule functionally equivalent to GRPO: a policy gradient term scaled by a baselined reward, plus an explicit regularization term pulling the policy toward the reference. The key differences from GRPO are that standard deviation normalization is absent (avoiding the GRPO bias on near-uniform reward groups) and the regularization is an explicit squared-loss penalty rather than clipping.

On top of this RL objective, Kimi adds a length reward to directly control CoT verbosity:

where $L_{\min}$ and $L_{\max}$ are the shortest and longest rollout lengths in the current batch, $| o |$ is the response length, and $c \in \{ 0 , 1 \}$ is the correctness indicator. Correct answers are rewarded for being short; incorrect answers are weakly penalized for being long.

Qwen 3 follows the R1 pipeline closely-SFT cold-start, reasoning RL, post-training RLHF-but adds thinking mode fusion: after RL, the model is fine-tuned via SFT to respond to a <think> tag with a full CoT and to a no_think tag with a direct answer. A single set of model weights serves both modes. Critically, mid-CoT generation can be interrupted by injecting a special prompt string, causing the model to emit its best current answer before the full reasoning budget is exhausted.

The two innovations that distinguish these papers from R1 are the length reward (Kimi) and dynamic inference budget control via thinking mode fusion (Qwen 3).

The Kimi length reward addresses the most practical objection to long-CoT reasoning models: inference cost. Without explicit length incentives, GRPO training creates a perverse incentive-when a rollout is incorrect and receives negative reward, the cheapest escape is to produce a longer response, diluting the per-token penalty under length-normalized objectives. The Kimi reward breaks this by making correct and short the uniquely rewarded combination, while incorrect responses are incentivized toward batch-average length rather than allowed to grow unboundedly. The reward must be introduced only after initial RL convergence; adding it at the start traps the model in a short-answer local minimum before it has learned to solve problems at all.

Qwen 3’s thinking mode fusion is a pragmatic response to the same problem from the serving side. Rather than training and deploying separate thinking and non-thinking models-doubling storage, serving infrastructure, and operational complexity-a single model can serve both modes. The mid-generation interrupt is particularly useful for latency-constrained applications: set a maximum thinking-token budget, inject the interrupt string at that boundary, and the model produces a valid answer based on partial reasoning rather than timing out or truncating incoherently. Qwen 3’s ablation confirms graceful degradation: performance drops smoothly as the thinking budget shrinks from unconstrained to zero, rather than falling off a cliff.

A Worked Example

Kimi length reward arithmetic. Suppose a training batch produces rollouts of lengths 512, 1,024, 2,048, and 4,096 tokens. Then $L_{\min} = 512$ and $L_{\max} = 4 , 096$.

• Correct answer at 512 tokens: $\lambda = 0 . 5 - ( 512 - 512 ) / ( 4096 - 512 ) = + 0 . 5$. Length reward $= + 0 . 5$.

• Correct answer at 4,096 tokens: $\lambda = 0 . 5 - ( 4096 - 512 ) / ( 4096 - 512 ) = - 0 . 5$. Length reward $= - 0 . 5$.

• Incorrect answer at 4,096 tokens: $\lambda = - 0 . 5$; since $c = 0$, $r_{\mathrm{len}} = \min ( - 0 . 5 , 0 ) = - 0 . 5$.

• Incorrect answer at 512 tokens: $\lambda = + 0 . 5$; since $c = 0$, $r_{\mathrm{len}} = \min ( + 0 . 5 , 0 ) = 0$.

The total reward is $r_{\mathrm{acc}} + r_{\mathrm{len}}$. A correct short answer gets $+ 1 . 5$; a correct long answer gets $+ 0 . 5$; an incorrect short answer gets $0$; an incorrect long answer gets $- 0 . 5$. The optimization pressure is unambiguous: be correct and concise.

Qwen 3 RL scale. The Qwen 3 RL stage uses just 3,995 training problems-a surprisingly small number. Despite this, the RL stage delivers measurable gains on math and STEM benchmarks relative to the SFT-only model. This illustrates that RLVR is highly sample-efficient on verifiable tasks: the model already knows much of the underlying reasoning; RL is refining the strategy for applying it.

Thinking mode fusion data construction. After the RL stage, construct two types of SFT examples:

• Thinking mode: <think> [long CoT from the RL model] </think> [final answer].

• Non-thinking mode: <no_think> [direct answer, no CoT]. Fine-tune the RL model on a mix of both. After this supervised stage, the model respects the tag semantics and, when generating in thinking mode, will respond coherently to the interrupt string by emitting an answer based on its partial reasoning.

What You Actually Decide in Practice

• Gate the length reward on only after RL stabilizes. Adding the Kimi length reward from step one forces the model into a short-answer local minimum before it has learned to solve anything. In practice, run unconstrained RL until reward curves show clear upward progress, then introduce the length reward. A rough heuristic: wait until mean accuracy reward on the training distribution exceeds 0.3.

• Use a learned judge for answer equivalence, not only regex. Kimi uses a fine-tuned reward model for math answer comparison rather than relying solely on SymPy or regex. Notation variants (e.g., $\frac{1}{2}$ vs. $0 . 5$ vs. 1/2) break naive string matching at scale. A model trained specifically on equivalence checking is more robust and can be made very accurate because the task is narrow.

• Separate inference and training workers. RL training requires alternating between rollout generation (inference) and parameter updates (training). Running both on the same GPUs causes memory thrashing and poor utilization. The standard architecture has a pool of inference workers running vLLM and a pool of training workers, with weight updates broadcast via NCCL or by restarting the inference server. Budget for this infrastructure; it is the hardest engineering component in reasoning model RL.

• Filter data by difficulty before RL. Both Kimi and Qwen 3 exclude problems where the untuned model can solve at least one of eight rollouts (pass rate $\geq 1 / 8$). Spending compute on problems the model already handles is waste. Continually re-evaluate the training set difficulty as the model improves and prune easy problems dynamically.

• Expect RLHF post-training to trade math for instruction following. Qwen 3’s ablation shows the general RLHF stage improves instruction-following and general tasks but slightly hurts math and STEM scores in thinking mode. This is not a bug-the RLHF objective does not reward math accuracy-but it is a trade-off to design for explicitly when building a multi-capability model.

How This Shows Up in Interviews

• “How would you control the inference cost of a reasoning model?” Strong answer: add a length reward during RL that incentivizes correct-and-short responses over correct-and-verbose ones; introduce it after initial convergence to avoid the short-answer local minimum. For serving, add thinking mode fusion so that operators can cap the thinking-token budget at inference time, getting graceful degradation rather than abrupt failure.

• “What are the infrastructure bottlenecks when training a reasoning model with RL?” Strong answer: three main bottlenecks. First, rollout generation blocks training; separate inference and training worker pools with asynchronous message passing. Second, variable CoT lengths create uneven batches and GPU underutilization; implement sequence packing or per-length-bucket batching. Third, syncing updated weights from the training process to the vLLM inference server is non-trivial; current approaches either use an experimental NCCL weight-loading API or tear down and restart vLLM each iteration.

• Senior follow-up: “Kimi K1.5 and DeepSeek R1 both work, but with different RL objectives. How do you choose between them?” Senior answer: both converge to the same structure-policy gradient scaled by a baselined reward, plus regularization toward a reference policy-and empirical differences between them are smaller than differences introduced by data quality and reward design. The more consequential choice is whether to add a length reward (Kimi has it; vanilla GRPO does not) and whether to use standard deviation normalization in the advantage (the Dr. GRPO analysis argues against it). In practice, start with GRPO for familiarity, add a length reward after initial convergence, and remove standard deviation normalization if you observe training instability on near-uniform reward groups.

Key Takeaways

• Kimi K1.5 derives its RL objective from the DPO optimality condition without pairwise preferences, arriving at a policy gradient update similar to GRPO but without standard deviation normalization.

• A length reward is essential to prevent unbounded CoT growth; add it only after initial RL convergence, not from the start, to avoid trapping the model in a short-answer local minimum.

• Qwen 3’s thinking mode fusion trains a single model to serve both thinking and non-thinking requests, enabling graceful performance degradation under a token budget at inference time.

• RLVR is highly sample-efficient: Qwen 3 achieves strong reasoning gains from fewer than 4,000 training problems.

• The dominant infrastructure challenge in reasoning model RL is weight synchronization between training and inference workers; allocate engineering time for this before starting large RL runs.