Chapter 26 Supervised Fine-Tuning

26.1 What SFT Can and Cannot Teach

26.1.1 Style, Format, and Task Specification

An interviewer at a frontier lab will often open an SFT discussion with a deliberately simple question: “GPT-3 and ChatGPT use essentially the same architecture trained on similar internet data-what does fine-tuning actually do?” The answer is more nuanced than most candidates expect, and getting it wrong signals that you have only ever used these models, not built them. At a mid-level this question probes whether you understand the separation between pre-training and post-training; at a senior level it probes whether you can reason about which desired behaviors SFT can deliver reliably and which it fundamentally cannot.

Pre-training packs the model with latent capabilities. Somewhere in a 7B parameter checkpoint trained on two trillion tokens lives the ability to summarize, translate, write code, and answer questions-but none of it fires reliably on a bare prompt like Summarize this article:. The model was never trained to respond to that framing; it was trained to predict the next token in a web-scale corpus. Supervised fine-tuning bridges that gap.

Mechanically, SFT is nothing more than continued gradient descent on a new dataset. You format each example as an instruction-response pair, concatenate them with a chat template, and compute cross-entropy loss only on the response tokens:

where $x$ is the instruction prefix (masked out of the loss) and $y_{1} , . . . , y_{T}$ is the target response. Nothing exotic happens algorithmically. The power comes from what the data teaches.

What SFT reliably teaches falls into three buckets. First, output format: whether to respond in prose, bullet points, markdown, code blocks, or structured JSON. Second, task framing: the model learns what kinds of completions belong to what kinds of prompts-that a prompt asking for a “subject line” expects a short noun phrase, not a paragraph. Third, response style: register (formal vs. casual), length norms, whether to ask clarifying questions, whether to hedge uncertainty. These are all properties of the output distribution’s shape-its “type signature,” to borrow a programming term.

SFT teaches the model what kind of token sequence to produce; whether the content of that sequence is factually correct or deeply reasoned depends almost entirely on what the base model already knows.

The intuition is distributional. A base model has seen instruction-response pairs in pre-training-Stack Overflow Q&A, documentation, tutorials-but they appear infrequently and inconsistently formatted. SFT concentrates that signal: every example in the fine-tuning set is an instruction followed by a high-quality completion, so the model learns a strong prior that this template triggers a particular response mode.

This matters for an important reason: the cost structure is favorable. You can dramatically change how a model presents information with far fewer examples than you might expect. Studies with early open models showed that fine-tuning on as few as 500 to 1,000 high-quality examples moved a base model’s output distribution much closer to a deployed assistant. The data quality multiplier is enormous at this scale-ten carefully written examples often outperform a thousand noisy ones.

The trade-off is that surface behavior and underlying capability can diverge. A model fine-tuned on data where every answer is formatted as a numbered list will produce numbered lists. It will do so even when the question calls for a single-sentence reply, and even when the underlying reasoning is absent or wrong. Style and correctness are trained independently. This is exactly why length and formatting biases are so persistent: human raters and LM judges both prefer longer, list-heavy responses, so any SFT data shaped by human or AI feedback will systematically drift toward that style regardless of whether more words mean better answers.

A Worked Example

Take a Llama-class 7B base model (d_model=4096, 32 layers, bfloat16, trained on roughly 2T tokens). Feed it the raw prompt:

Explain the difference between precision and recall.

The base model might continue with something like “Precision and recall are two metrics commonly used in... The precision of a binary classifier is defined as...” followed by a rambling continuation that drifts into unrelated content, because the base model is completing a document, not answering a question.

After SFT on 50,000 instruction-response pairs with 3 epochs, batch size 128, learning rate $2 \times 10^{- 5}$

with cosine decay (total compute roughly 150 GPU-hours on 8 $\times$ H100s), the same prompt now elicits a clean structured answer: a two-sentence definition of each metric, a concrete example with a confusion matrix, and a note on when to prioritize each-because hundreds of similar prompt-response pairs in the training set established that template.

What changed? Not the underlying knowledge of precision and recall-that was already in the base model. What changed is the trigger: the model now recognizes a standalone instructional prompt as a signal to enter “teaching assistant” mode, adopt an explanatory register, and terminate with a clean conclusion. The computation that produces the correct definition is largely reusing pre-trained circuits; SFT just learned the routing.

The same experiment illustrates the format ceiling. If you now ask the same SFT model to solve a novel multi-step math problem the base model never mastered, the SFT model will confidently produce a wellformatted wrong answer. The format is right; the substance is not. SFT cannot manufacture capability that does not exist in the pre-trained weights.

What You Actually Decide in Practice

• Volume vs. quality threshold. Below roughly 1,000 examples, quality dominates; every noisy example noticeably hurts. Above 100,000, you can tolerate some noise and diversity starts to matter more than per-example quality. If you are resource-constrained, invest in curation, not volume.

• Template coverage over domain coverage. It is more important that your SFT data covers the space of interaction patterns (multi-turn, tool-calling, refusal, clarification) than that it covers every knowledge domain. The base model handles domain breadth; SFT handles behavioral patterns.

• Fight length bias explicitly. Human raters prefer responses approximately 60-70% of the time when those responses are longer, all else equal. LM judges share this bias. If you use human or AI preference signals to curate SFT data, audit response lengths and actively include short, correct answers. Failing to do so produces a model that hedges everything and adds unnecessary preamble.

• Safety data must be mixed in, not applied separately. A model SFT’d purely on capability demonstrations and then separately safety-tuned often exhibits over-refusal-it refuses borderline but safe queries like “how do I kill a Python process?” because the safety tuning had no context about the capability distribution. Mixing even 500 carefully crafted safety examples into the main SFT run dramatically reduces this problem.

• Modern pipelines blur pre-training and SFT. At frontier scale, instruction data is increasingly injected during the learning-rate decay phase of pre-training (“mid-training”), not saved for a separate SFT pass. This avoids catastrophic forgetting of pre-trained capabilities, gives the instruction data more gradient steps, and allows instruction-tuning data volumes that would be prohibitively expensive in a standalone SFT stage.

How This Shows Up in Interviews

• “What does SFT actually change in a pre-trained model?” Strong answer: SFT reshapes the output distribution’s surface properties-format, register, task framing-without adding new world knowledge. The base model has latent capabilities; SFT installs the routing that triggers them on instructional prompts. Mention that the effect is large even with small data, and that quality dominates quantity at this scale.

• “Why does SFT improve benchmark performance even with very little data?” Strong answer: Benchmarks like MMLU are multiple-choice or short-answer; the format is simple and the knowledge is already in the base model. SFT teaches the model to emit the expected output format, which is almost all that is needed. The jump from base to instruction-tuned model on MMLU is largely a formatting effect, not a knowledge effect.

• “A model fine-tuned on our data produces outputs that are always too long. How do you fix it?” Strong answer: Audit your SFT data for length distribution and human/AI curation bias. Explicitly add short-answer examples. If using preference data to filter SFT examples, control for length when computing preference scores. Do not just add a system prompt instructing brevity-that is a band-aid.

• Senior follow-up: “How would you approach SFT if your instruction data was generated by a stronger teacher model like GPT-4?” The deeper concern is distribution mismatch: the teacher’s responses may require knowledge or reasoning ability the student model does not yet have. If the student must produce tokens it cannot faithfully generate from its own weights, it learns to imitate the structure of the response while potentially hallucinating the content. The mitigation is to filter teacher-generated data to only include examples where the student, when probed, already demonstrates the underlying knowledge-effectively conditioning SFT data on what the student can support.

Key Takeaways

• SFT reliably teaches output shape: format, style, task framing, and interaction patterns. It does not reliably teach new factual knowledge.

• The effect is large even with small data; quality dominates quantity below roughly 10,000 examples.

• Length and list-formatting biases are structural artifacts of human and AI preference data; actively audit and counteract them.

• Modern pipelines inject instruction data during the decay phase of pre-training (mid-training) to scale up without catastrophic forgetting.

• SFT cannot manufacture capability the base model does not possess; it can only route and express what is already there.

26.1.2 Risks of Teaching Hallucination

When a candidate in a systems interview says “we just need higher-quality SFT data,” the follow-up that separates senior engineers from mid-level ones is: “what does high quality actually mean, and can SFT data ever make a model worse?” The answer is yes, and the failure mode is counterintuitive-fully correct, richly detailed SFT examples can actively train a model to fabricate. This is one of the most consequential and least understood pitfalls in post-training pipelines, and it surfaces in staff-level interviews whenever the conversation turns to hallucination, distillation, or why RL is needed at all.

Supervised fine-tuning is next-token prediction on a fixed dataset. Every token in the target response contributes to the loss, and the model has no mechanism to distinguish between tokens it can faithfully generate from its own weights and tokens it is being forced to predict without supporting knowledge. The cross-entropy loss treats all target tokens symmetrically:

This symmetry is the source of the problem. Suppose the target response contains a factual claim or citation that the pre-trained model does not reliably know. To minimize the loss, the model faces two competing mechanisms. The first is genuine knowledge retrieval: if the model already encodes the relevant fact in its parameters, it learns to surface that fact in the right context-a good outcome. The second is structural imitation: the model learns that completions with a certain input structure (“write an introduction to a complex economics concept”) should end with a reference, and it learns to produce tokens that look like a citation regardless of whether those tokens are grounded in anything. Structural imitation is always easier to learn than knowledge retrieval, because structure is a shallower pattern and the gradient signal for it is denser.

The key insight is that from the model’s perspective, hallucinating a plausible-looking citation incurs less loss than producing nothing at the end. The structure of the response must be completed. If the model lacks the parametric knowledge to fill the slot correctly, it fills the slot with invented content because that is the lower-loss option. SFT does not teach the model to abstain; it teaches the model to complete.

The deeper a model’s SFT data exceeds the model’s own knowledge, the stronger the incentive to learn structural imitation over knowledge retrieval-and the worse the hallucination problem becomes.

The standard intuition for SFT data quality is “more knowledge, longer responses, more citations is better.” That intuition is wrong for a model that has not already internalized the underlying facts. Consider two explanations for what happens when you fine-tune on a response that cites a specific economics paper: (a) the model learns to associate that input topic with that specific paper, surfacing it reliably at inference time; or (b) the model learns the abstract behavioral pattern “complex concept $\rightarrow$ response $\rightarrow$ citation.” If the model genuinely knows the citation from pre-training, (a) dominates. If it does not, (b) dominates-and (b) generalizes in the worst possible way: it fires on every complex question and fills the citation slot with whatever token sequence has the highest probability under the language model, typically a plausible-but-fabricated author-year pair.

The contrast with what RL-based methods do is instructive. Reinforcement learning with verifiable or preference-based rewards provides a signal that is conditional on whether the model got something right. SFT provides no such signal-it applies gradient descent unconditionally. This is why on-policy RL is a principled remedy: you only reinforce completions the model itself produced and got correct, so you never update on knowledge the model does not possess.

The same failure mode applies to knowledge distillation. When a small student model is fine-tuned on outputs from a large teacher model, the teacher’s responses will frequently describe facts, reasoning chains, or technical details that are simply not encoded in the student’s smaller weights. The student cannot distinguish “I should learn this fact” from “I should learn to produce tokens that look like I know this fact.” At small model scales, the distillation signal that appears to improve benchmark performance is often actually training the student to pattern-match the teacher’s output structure.

A Worked Example

Consider fine-tuning a 7B model (d_model=4096, 32 layers, bfloat16, pre-trained on 1T tokens) on 50,000 instruction-response pairs drawn from an OpenAssistant-style dataset. A representative example in the dataset:

Instruction: Write an introduction to monopsony in labor economics.

Response: Monopsony occurs when a single employer dominates a labor
market, giving it wage-setting power over workers. Bivens and Mishel
(2013) document how this mechanism has suppressed wage growth in the
United States over the past three decades, estimating that monopsony
rents account for roughly 25% of the gap between productivity and
compensation since 1979. [Bivens, J. and Mishel, L. (2013). ...]

After 3 epochs of SFT, two things have happened simultaneously. First, if “Bivens and Mishel (2013)” appeared in the pre-training corpus with sufficient frequency, the model may have strengthened its association between “monopsony,” “labor economics,” and that specific citation-a genuine knowledge reinforcement. Second, and more likely for a 7B model trained on 1T tokens where this specific paper may appear only a handful of times, the model has learned the pattern: [economics topic] $\rightarrow$ [prose introduction] $\rightarrow$ [Author Year (Year) summary sentence]. Now run inference on a different economics question:

Instruction: Write an introduction to oligopsony in shipping markets.

The model produces a fluent, well-formatted response ending with a fabricated citation-say, “Jensen and Kowalski (2017)”-because the citation slot must be filled and the most probable tokens for that slot given the structural pattern are a plausible author-year pair. The model did not retrieve; it completed a template.

To verify this at development time: before adding citation-heavy examples to your SFT set, probe the base model with the underlying claims in isolation. If the model cannot reliably reproduce “Bivens and Mishel documented monopsony rents” in a zero-shot probe, that SFT example will train structural imitation, not knowledge retrieval. Filter it out, or replace the citation with “according to labor economists” to keep the useful instruction-following signal without encoding a specific fabrication pattern.

What You Actually Decide in Practice

• Probe before you fine-tune. Before including a factual claim or citation in SFT data, test whether the base model can produce that fact unprompted. A simple consistency check-ask the same underlying question five times and measure agreement-takes under an hour on a batch of examples and filters out the most dangerous cases. Examples where the model is inconsistent or confidently wrong are the highest-risk SFT candidates.

• Prefer abstention patterns over forced answers. If you cannot verify that the model knows a fact, include a paired abstention example: the same input, with a response that says “I don’t have reliable information on this specific point.” Training models to abstain gracefully is far more valuable than training them to answer confidently and wrong. Even 100-200 abstention examples mixed into a 50,000-example SFT set noticeably reduces fabrication rates on out-of-distribution queries.

• Cap teacher-student capability gap. When generating SFT data from a stronger teacher model (e.g., using a 70B or GPT-4-class model to write responses for a 7B student), restrict examples to tasks the student can plausibly learn. A practical heuristic: if the student’s loss on the teacher response exceeds a threshold (typically 3-4 nats per token on held-out examples), that example is likely training structural imitation and should be dropped or simplified.

• Separate knowledge from format signals. If the goal is to teach the model a new response format (chain-of-thought, JSON, structured summaries), use examples where the content is simple and already well within the model’s knowledge. Combining novel format with novel content in the same examples conflates two learning signals and increases hallucination risk. Teach format first on known content, then introduce harder content once the format is stable.

• SFT cannot fully solve this; RL is the complement. No amount of SFT data curation perfectly eliminates hallucination, because you can never fully audit which facts a model “knows reliably.” On-policy RL with a reward that penalizes factual errors-or even simple best-of-N rejection sampling-provides the adaptive per-example correctness signal that SFT structurally cannot. Treat SFT as installing behavioral patterns, and RL as correcting factual reliability on top of those patterns.

How This Shows Up in Interviews

• “Why does fine-tuning sometimes increase hallucination rates compared to the base model?” Strong answer: SFT applies loss unconditionally on all target tokens. When the training data contains claims the model does not encode, the model minimizes loss by learning to structurally imitate the response format rather than retrieve the underlying knowledge. This is cheapest in loss terms and generalizes badly-the model learns to confidently complete response templates even without grounding.

• “You are building a medical QA assistant using SFT on doctor-written responses. What data quality checks would you run?” Strong answer: For each doctor-written response, probe the base model for the underlying factual claims before including the example. Filter examples where the base model is inconsistent or fabricates on similar queries. Include explicit abstention examples for edge cases. Consider a lower-fidelity but factually safe alternative: simplify cited claims to general attributions, rather than training the model to pattern-match specific author-year pairs it cannot verify.

• “What is the relationship between SFT data quality and hallucination, and how does it motivate RLHF?” Strong answer: High factual depth in SFT data is only safe when the model already possesses the underlying knowledge. If there is a capability gap between the data and the model, SFT teaches structural completion rather than knowledge retrieval. RLHF provides a correctness-conditional signal: in PPO, rewards are only propagated for completions the model itself generates and that are judged correct, so the model never takes gradient steps on knowledge it does not have.

• Senior follow-up: “How would you design a data curation pipeline that automatically identifies SFT examples likely to cause hallucination, at the scale of millions of examples?” The senior answer involves a self-consistency filter: for each candidate SFT example, sample $k$ completions from the base model on the instruction alone (without the reference response). Compute agreement across samples using a lightweight verifier or embedding similarity. Examples where base model completions show high agreement with the reference response are safe; examples where the base model is inconsistent or produces completions that contradict the reference are flagged as hallucination-risk examples. At scale, run this filter on GPU with small-$k$ sampling (say, $k = 5$) to keep cost manageable-this adds roughly 5$\times$ the inference cost of the SFT dataset but is far cheaper than dealing with hallucinating deployed models.

Key Takeaways

• SFT loss is unconditional: the model takes gradient steps on every target token regardless of whether it has the underlying knowledge, creating an incentive to learn structural imitation over factual retrieval.

• Fully correct, richly detailed SFT data can increase hallucination when the model does not already possess the referenced facts-counterintuitive but well-documented.

• The capability gap between teacher and student in distillation amplifies this risk; filter or simplify examples where the student’s per-token loss on the teacher response is abnormally high.

• Mixing abstention examples (“I don’t know”) into SFT data is one of the few tractable SFT-level mitigations; RL with correctness-conditional rewards is the principled long-term fix.

• At interview, connect this failure mode directly to the motivation for RLHF: RL’s on-policy correctness signal is precisely what SFT lacks.

26.2 SFT Data in Practice

26.2.1 Aggregated, Human-Written, and Model-Generated Data

When a team puts you in charge of building the SFT pipeline for a new model, the first concrete question is not which optimizer to use - it is where the training examples come from. Three distinct paradigms have emerged across the open literature, each with a different cost profile and a different failure mode. Interviewers at frontier labs ask about this because the data composition decision is real, made early, and has outsized downstream consequences. Expect it at mid-to-senior level when the conversation shifts from “how does SFT work” to “how would you actually build it.”

The first paradigm is aggregated NLP data. The idea is to take existing supervised benchmark datasets - summarization, question answering, classification, slot filling - and reformat each example as a natural-language instruction followed by the expected output. A summarization pair becomes “Write highlights for the following article: [article]” and a classification pair becomes “What topic does this text cover? Options: business, sports, politics.” The payoff is that you can assemble millions of examples at near-zero marginal cost. The cost is that the resulting distribution looks nothing like a real chat interface: inputs are short and formulaic, outputs are often single words or phrases, and the whole dataset retains the fingerprints of benchmark surgery. FLAN is the canonical example of this approach, aggregating dozens of NLP tasks into one large metadata set.

The second paradigm is human-written crowdsourced data. Human annotators write both the instruction and the response from scratch. When motivated and given enough time, they produce diverse, natural, high-quality examples. The scaling problem is severe. Writing a thorough, accurate, multi-paragraph response to a nuanced question takes ten to fifteen minutes for a skilled annotator. At $0.10-$0.50 per example, 100,000 examples costs $10,000-$50,000 and still may not cover the tails of the prompt distribution. In practice, time-pressured annotators produce short, low-effort completions; the long, expert-quality responses in any crowdsourced dataset tend to trace back either to domain experts or to AI copy-paste. OpenAssistant is the clearest public example of this paradigm’s ceiling and its floor.

The third paradigm is model-generated data. A seed set of a few hundred human-written instructions is used to prompt a capable model to generate both new instructions and their responses. Stanford Alpaca demonstrated this at scale: GPT-3 completions filled in 52,000 instruction-response pairs at a cost under $500. The trade-off is that the student model learns the teacher’s output distribution, including any hallucination tendencies, and the diversity of the seed set limits the diversity of the generated set.

SFT data paradigm choice is not a one-time decision but a mixing problem: each paradigm fills a different gap in quality, cost, and coverage.

No single paradigm dominates on all axes. Aggregated NLP data wins on volume and is excellent for breadth across structured tasks, but models trained on it consistently underperform on open-ended chat evaluations because they learn a benchmark-centric output style - short, formulaic, task-conditioned rather than conversationally grounded.

Human-written data wins on naturalness and, when collected carefully, on factual accuracy. The problem is scale. A well-run annotation campaign at a frontier lab might produce 100,000-200,000 examples. That is enough to establish a strong behavioral baseline, but not enough to cover the long tail of real-world prompts. The gap between what motivated annotators write and what time-pressured crowdworkers produce is also wider than most engineers expect when they first run a labeling project.

Model-generated data has become the workhorse of modern SFT pipelines because it occupies the right trade-off: cheap, naturally phrased, and scalable to tens of millions of examples. The failure mode to manage is hallucination transfer. When the teacher model is significantly stronger than the student, the student may learn to mimic the surface form of confident, detailed responses without having the underlying factual knowledge to support them. The practical rule: keep teacher and student within roughly two to three capability levels of each other, and audit generated responses for claims the student model could not have seen during pre-training.

The modern synthesis is mid-training: mixing SFT examples directly into the learning-rate decay phase of pre-training rather than treating SFT as a separate post-training step. This sidesteps catastrophic forgetting and allows the instruction-following signal to be learned at a deeper representational level.

A Worked Example

MiniCPM, an openly documented small-model series, illustrates the two-stage approach with concrete data. Their training uses a Warm-up, Stable, Decay (WSD) learning-rate schedule totaling approximately $4 \times 10^{12}$

tokens.

During the stable phase the data mix is entirely pre-training corpora: Common Crawl, code, the Pile, and Dolma. During the decay phase - roughly the final 10% of total compute - the mix shifts substantially:

• Wikipedia and high-quality curated web text (approximately 30% of the decay mix)

• Instruction-following data: UltraChat, Evol-Instruct, OSS-Instruct (approximately 30%)

• Code SFT and domain-specific Q&A: Stack Exchange, structured coding exercises (approximately 20%)

• Continued general pre-training data to prevent catastrophic forgetting (approximately 20%)

The training loss drops sharply at the start of the decay phase, which practitioners interpret as the model rapidly consolidating the instruction-following signal as the learning rate falls. After the decay phase, a short conventional SFT run of a few thousand steps on the highest-quality subset reinforces instruction-following behavior. Ablations from this work show that models trained with the two-stage approach outperform models fine-tuned only at the end, particularly on tasks requiring factual recall combined with instruction following - exactly the regime where late-stage SFT risks catastrophic forgetting.

What You Actually Decide in Practice

• Validate mix ratios before committing to scale. Build a 50,000-100,000 example pilot from all three paradigms and run evals before assembling a million-example pipeline. Relative proportions matter more than total volume past a few hundred thousand examples.

• Length bias is a real confounder. Both human raters and LM-as-judge systems prefer longer outputs roughly 60-70% of the time independent of content quality. Never use chatbot-arena-style win rates as your sole quality signal. Cross-check on capability benchmarks like MMLU or GSM8K where response length cannot inflate scores.

• SFT teaches format reliably; it does not teach new facts reliably. Fine-tuning will teach a model to always respond in bullet points, to open with a summary sentence, or to append a disclaimer. It will not reliably teach a model a fact absent from pre-training. SFT data containing citations or specific statistics the model has never seen will produce the format of confident citation without the correct underlying knowledge.

• Crowdsourcing quality requires explicit incentive design. Paying per example produces short examples. Audit a random sample of every batch; set a minimum substantive length that requires genuine content, not padding. Expect that the longest, most detailed responses in any crowdsourced set were written by a small fraction of high-effort annotators or generated by AI and submitted directly.

• Mid-training is worth the infrastructure cost at scale. If you have more than approximately 1 million SFT examples and a pre-training pipeline already in place, mixing into the decay phase almost always outperforms late-stage SFT on chat benchmarks and shows fewer signs of capability degradation.

How This Shows Up in Interviews

• “Walk me through the trade-offs between FLAN-style aggregated data and model-generated data for instruction tuning.” Strong answer: FLAN is cheap and broad but unnatural, producing benchmarkcentric output styles; model-generated data is more naturally phrased and scalable but risks hallucination transfer when the teacher is much stronger than the student. Both tend to underperform human-written data on naturalness but scale better on volume.

• “Why might a model trained on high-quality, expert-level SFT data hallucinate more than one trained on shorter, simpler responses?” Strong answer: when SFT examples contain knowledge the model does not have from pre-training, the model learns the output structure - confident, detailed, cited - without the factual content. It fills the structural template with plausible-sounding fabrications because the loss penalizes wrong tokens more than it rewards honesty about uncertainty.

• “You have a limited annotation budget of $50,000. How do you allocate it across data paradigms for a 7B model SFT run?” Strong answer: use model-generated data from a strong API model for breadth (500,000-1,000,000 examples at near-zero marginal cost), reserve the human budget for high-value borderline and safety examples (5,000-10,000 carefully reviewed pairs), and fill structured-task coverage gaps with aggregated NLP data.

• Senior follow-up: “How would you detect and correct for length bias in your SFT evaluation pipeline?” Senior answer: disaggregate win rates by output-length bracket; run a reward model trained explicitly to be length-neutral; compare chatbot-arena rankings against benchmark rankings and investigate divergences; introduce length-controlled paired comparisons where both responses are truncated to the same token count before human or model rating.

Key Takeaways

• Three paradigms exist - aggregated NLP, crowdsourced human-written, and model-generated - each trading cost, naturalness, and scalability differently; modern pipelines blend all three.

• SFT reliably shapes output format and style; teaching new factual knowledge requires that knowledge to already exist in pre-training.

• Length bias inflates both human and LM-judge preference scores by 60-70%; always cross-validate against capability benchmarks that cannot be gamed by longer outputs.

• Mid-training (mixing SFT data into the learning-rate decay phase) is the current state-of-practice at scale and largely avoids catastrophic forgetting.

• Crowdsourcing quality degrades sharply under time pressure; incentive design and batch auditing are as important as vendor selection.

26.2.2 Safety Tuning and Refusal Trade-Offs

Every LLM that ships to users has to answer a practical question: what does the model do when someone asks it to do something it should not do? The naive implementation is to collect harmful prompts and train the model to output refusals. The problem is that a model trained only on refusals develops the wrong decision boundary - it pattern-matches on surface features like the word “kill” and refuses “how do I kill a Python process” alongside genuinely harmful requests. This failure mode is as important to understand as hallucination, and it shows up in both early-stage design questions (“what is safety tuning?”) and senior-level architecture discussions (“build me the safety data pipeline for a production model”).

Safety tuning is a targeted SFT procedure that teaches the model two distinct behaviors: when to refuse, and how to do so gracefully without lecturing or over-explaining. The training data comes in two structurally critical categories.

Harmful examples are (prompt, refusal) pairs where the prompt requests something clearly off-limits: detailed instructions for generating scam content, synthesis routes for dangerous materials, targeted harassment of a specific individual. The completion is a polite, brief refusal that acknowledges the request type without restating it in detail.

Borderline-positive examples are (prompt, compliant response) pairs where the prompt superficially resembles a harmful request but is entirely legitimate: a security researcher asking about exploit mechanics, a physician asking about lethal medication doses for treatment planning, a sysadmin asking how to terminate a process. These examples define the decision boundary from the other side. Without them, the model’s refusal behavior drifts toward surface features of the harmful examples rather than their semantic intent.

The informal but important principle is: for any vocabulary cluster $V$ that appears prominently in harmful prompts, the training data needs at least as many examples of legitimate uses of $V$ as harmful ones, or the model learns $V \Rightarrow$ refuse rather than harmful_intent$( V ) \Rightarrow$ refuse.

A safety-tuned model that over-refuses is not safe - it is broken in a different direction, and the product experience consequences are just as real.

The finding that consistently surprises engineers new to post-training is how little data safety tuning needs. Research across multiple groups has shown that even 500-1,000 safety examples mixed into a standard SFT run produce a measurably safer model without meaningfully degrading capability benchmarks. The reason is that the pre-trained model already encodes a strong representation of harm, danger, and ethics from the pre-training corpus. Safety tuning does not teach the model what harm is - it teaches the model to act on that representation by refusing rather than complying.

This is the same leverage principle that makes SFT powerful in general: capabilities are baked in during pre-training; post-training steers the output mode. The corollary is equally important: safety tuning cannot fundamentally fix a model that has deeply harmful behaviors embedded in its pre-training. It can redirect surface outputs, not reorganize deep representations.

The over-refusal failure mode is symmetric to hallucination. In both cases the model defaults to a learned surface pattern rather than grounding its response in the actual prompt content. Hallucination fills the response template with invented facts; over-refusal fills the response template with a refusal pattern triggered by surface keywords. Both are the same underlying problem: the model learned a shortcut that satisfies the type signature of the training data without performing the intended reasoning.

A Worked Example

Suppose you are fine-tuning a 7B parameter model on a 50,000-example general SFT corpus. You have a budget for 1,000 additional safety examples. Consider two allocation strategies.

Strategy A uses all 1,000 as harmful-refuse pairs, covering topics such as fraud, dangerous synthesis, and harassment.

Strategy B splits the budget: 500 harmful-refuse pairs plus 500 borderline-positive pairs using the same vocabulary in legitimate contexts (technical jargon, medical queries, security research framing).

Evaluate on three metrics before and after each strategy:

1. Refusal recall: fraction of genuinely harmful prompts that receive a refusal. Target: above 95%.

2. Refusal false-positive rate: fraction of benign domain-specific prompts that receive a spurious refusal. Target: below 3%.

3. MMLU accuracy: should not degrade more than 0.5 absolute points from baseline SFT.

With Strategy A a typical 7B model achieves refusal recall of approximately 97% but a false-positive rate of 12-18%: the model learned to refuse any prompt containing vocabulary associated with harmful categories. With Strategy B recall stays at 96-97% and the false-positive rate falls to 2-4%. MMLU degrades by less than 0.3 points in both cases. The composition of the borderline-positive set is what drives the improvement: examples must share the same surface vocabulary as the harmful set but carry clearly legitimate framing, forcing the model to attend to framing rather than vocabulary alone.

What You Actually Decide in Practice

• Start at 1-2% safety data in the SFT mix. If your overall SFT corpus is 500,000 examples, start with 5,000-10,000 safety examples. Going above 5% of the total mix begins to noticeably degrade helpfulness and general-capability benchmarks without proportional safety gains.

• Match borderline-positive examples to harmful examples topic by topic. For every category of harmful prompts you cover - synthesis instructions, fraud scripts, targeted harassment - write an equal number of borderline-positive examples that use the same surface vocabulary in a legitimate context. Without this pairing, your model will over-refuse entire topic areas rather than specific intents.

• Refuse at the intent level, not the topic level. A topic-level rule such as “always refuse chemistry questions” will break legitimate medical and research use cases and will visibly frustrate users. Train the model to identify malicious intent within a topic, not the topic itself. This requires examples that make the distinction explicit, often with a brief explanation of why the request is complied with.

• Track refusal recall and false-positive rate on separate dashboard rows. An aggregate safety score that combines recall and precision hides over-refusal regressions behind high recall numbers. A model that refuses 99% of harmful prompts but also refuses 15% of benign technical queries is not a good model. Both numbers need to be visible and gated on.

• Safety properties degrade with continued post-training. Every subsequent SFT or RLHF round can erode safety behavior if safety data is not re-included in proportion. Schedule red-team evaluations after every major post-training update, not only at initial release. Safety is not a one-time certification.

How This Shows Up in Interviews

• “How do you build safety tuning data and what is the most common mistake?” Strong answer: collect both harmful-refuse pairs and borderline-positive pairs in roughly equal numbers. The most common mistake is collecting only harmful examples, which teaches the model to pattern-match on surface vocabulary rather than infer intent, causing high false-positive refusal rates on legitimate technical or domain queries.

• “A user reports that the model refuses ‘how do I kill a Python process.’ Walk me through diagnosing and fixing this.” Strong answer: (1) confirm the issue is reproducible on a held-out eval set; (2) inspect the safety training data for over-representation of the vocabulary cluster around “kill” in harmful-refuse pairs; (3) add borderline-positive examples using the same technical vocabulary in clearly legitimate contexts; (4) re-evaluate refusal recall to confirm the fix does not degrade safety recall; (5) add the repro case to the false-positive eval set so regression is detected automatically in future runs.

• Senior follow-up: “Design an eval suite that simultaneously detects regressions in safety and helpfulness after any data update.” Senior answer: maintain a fixed red-team set of approximately 500 harmful prompts measuring recall and a fixed borderline set of approximately 500 benign-but-surfacerisky prompts measuring false-positive rate; gate model releases on both passing defined thresholds; include MMLU and a chat-style benchmark to catch capability regressions; track all four metrics on a single release dashboard so no individual regression is absorbed by an aggregate score passing.

Key Takeaways

• Even 500-1,000 safety examples produce measurable impact; the pre-trained model already encodes harm representations - safety SFT steers the output mode, not the underlying knowledge.

• Borderline-positive examples are as important as harmful examples; omitting them causes topic-level over-refusal rather than intent-level refusal.

• Track refusal recall and refusal false-positive rate independently; aggregate safety scores hide over-refusal regressions behind high recall numbers.

• Keep safety data at 1-2% of the SFT mix; above 5% measurably harms helpfulness without proportional safety gains.

• Safety behavior degrades with continued post-training; schedule red-team evaluations after every major update, not only at initial release.