Chapter 23 Designing an Evaluation

23.1 Goals of Evaluation

23.1.1 Evaluating Models vs. Systems vs. Methods

You are asked in an interview to design an evaluation for a new LLM-powered coding assistant. Do you benchmark the underlying model weights, the full product stack, or the novel training recipe your team just shipped? These are not the same question, and confusing them is one of the most common evaluation mistakes I see from engineers at all levels. Getting the object of evaluation wrong means the number you report answers a different question than the one you are trying to answer-and that mismatch quietly propagates into bad product decisions and misleading research claims.

There are three distinct objects you can evaluate, and each requires a different setup.

A model evaluation fixes the neural network weights and a minimal, standardized inference call: no external tools, no retrieval pipeline, no prompt engineering beyond what every competitor also gets. You are asking: what can these weights do on their own? The input is a prompt; the output is a completion; the metric is computed directly on that completion. This is what leaderboards like MMLU, GPQA, and HumanEval measure.

A system evaluation treats the entire production stack as the unit under test-model weights, prompt templates, RAG pipeline, tool calls, re-rankers, safety filters, and any agent scaffolding that drives multi-step loops. The user does not care which specific checkpoint sits at the center; they care whether the product solves their problem. System evaluations answer the question a buyer or a product manager is actually asking.

A method evaluation is what a research paper should be doing. Here the claim is that some algorithmic change-a new attention variant, a different data mixture, a revised RLHF loss-causes an improvement. The model trained with the new method is just a vehicle for demonstrating that claim. A valid method evaluation requires an ablation that isolates the variable: same data, same compute budget, same architecture except for the proposed change. Without that control, you are evaluating a system or a model, not a method.

Every evaluation you design should start with one sentence that names which of these three objects you are measuring-because that sentence determines everything else.

The distinction matters because the three objects differ in what they hold constant. Model evals hold constant the inference protocol, so the score is portable: you can compare your checkpoint against anyone else’s checkpoint on the same benchmark. System evals hold constant the task and the user population, so the score is predictive of production outcomes. Method evals hold constant the training regime except for the proposed change, so the score is attributable to the hypothesis you are testing.

Mixing these up has real consequences. If you compare System A-a retrieval-augmented pipeline built on a mid-tier model-against a bare-model eval of System B’s underlying weights, you are largely measuring the quality of System A’s scaffolding. I have seen this comparison made in competitive benchmarking slides, and it always favors whoever built the better wrapper. Conversely, a method paper that reports only a system score (“our model plus chain-of-thought plus self-consistency reaches X%”) cannot be credited with the improvement because the chain-of-thought and self-consistency would lift any baseline similarly.

The naive alternative-just report the highest number the model achieves under any configuration-is tempting because it looks good in a paper or press release. The problem is that it makes the number uninterpretable to anyone trying to replicate or compare.

A Worked Example

Consider evaluating a 70B model on SWEBench, a benchmark where an agent is given a GitHub issue and a codebase and must produce a patch that makes the unit tests pass.

Configuration 1: model eval. You call the model once per issue with a zero-shot prompt containing the issue description and the relevant file contents. No tool calls, no multi-step loop, no linting pass. A strong 70B model achieves roughly 10-15% resolve rate in this configuration.

Configuration 2: system eval. You wrap the same checkpoint in a scaffolded agent: the agent reads files, runs bash commands, iterates on failing tests, and submits only when lint passes. The same 70B model now achieves 35-45% resolve rate-a two-to-three-times lift from scaffolding alone.

Configuration 3: method eval. Your research question is whether adding a “verify-before-submit” step (the agent runs the relevant unit tests before submitting the patch) improves resolve rate. You train two checkpoints with identical data and compute, differing only in whether verification rewards were included during RLHF. You evaluate both under the same scaffolded system from Configuration 2, with the verification step disabled at inference time for both models. If the checkpoint trained with verification rewards achieves 40% versus 36% for the baseline under identical scaffolding, you can attribute the 4-point delta to the training method.

Note what breaks if you skip the discipline: if you enable the verification step for the new model but not the baseline, the 4-point delta is confounded-you have no idea whether the improvement comes from the training change or the inference-time verification.

What You Actually Decide in Practice

Choose model eval when reporting public benchmarks or comparing against published checkpoints. Standardized prompting (zero-shot or fixed few-shot, no tools) keeps the playing field level. If your eval protocol differs from the benchmark’s canonical setup, report both numbers and explain the delta.
Choose system eval when making a deployment or procurement decision. A 5-point gain on MMLU from a new checkpoint means nothing if your production system uses RAG and the bottleneck is retrieval recall, not model knowledge. Evaluate the full pipeline on a traffic sample from your actual users.
Choose method eval when writing a research paper or justifying a training investment. Fix compute, data, and architecture except for the single variable under study. Reporting system improvements without ablations is a red flag reviewers will catch.
Watch for scaffolding budget asymmetry. In agent benchmarks, “number of model calls allowed” and “maximum context length” are parameters that can easily be tuned per model. If System A gets 50 toolcall turns and System B gets 20, you are measuring budget allocation, not capability. Always match scaffolding budgets when comparing systems.
Perplexity is closer to a pure model eval than task accuracy. Perplexity assigns credit at every token, does not depend on agent scaffolding, and is difficult to game through prompt engineering. For tracking model quality during pre-training, it is the right signal. For production decisions, it is not sufficient on its own-a model with lower perplexity on web text can still underperform on your specific downstream task.

How This Shows Up in Interviews

“How would you evaluate your company’s new code generation assistant?” Strong answer: distinguish the model eval (does the checkpoint improve on HumanEval or SWEBench compared to the previous checkpoint?) from the system eval (does the full product increase the fraction of user sessions where the user accepts the suggestion?). Explain that you need both: the model eval guides training; the system eval guides the deployment decision.
“A paper claims their new attention variant improves MMLU by 3 points. How would you assess whether that claim is valid?” Strong answer: check whether compute and data are matched, whether the baseline uses the same training recipe except for the attention change, and whether the improvement holds across multiple seeds. A 3-point gain that disappears under controlled ablation is a system variation (e.g., the new attention allowed a larger batch size), not a method result.
Senior follow-up: “Your system eval shows a 5% win rate gain over the previous release. Your VP wants to know whether to credit the new model checkpoint or the prompt engineering rewrite your team shipped at the same time. How do you disentangle them?” Senior answer: run a $2 \times 2$ factorialold model with old prompt vs. old model with new prompt vs. new model with old prompt vs. new model with new prompt-on a held-out traffic slice. The interaction term tells you whether the prompt and model improvements are additive or whether one dominates. If you cannot run the factorial (the old model is decommissioned), you have a confounded result and should say so explicitly rather than attributing the gain to the most recently shipped change.

Key Takeaways

A model eval measures weights under a standardized inference call; a system eval measures the full production stack; a method eval isolates a single training variable under controlled ablation. Name which one you are running before you design anything else.
Comparing a scaffolded system against a bare-model baseline is not a model comparison-it is a scaffolding comparison. Match inference protocols when comparing models.
Agent benchmarks like SWEBench can produce a two-to-three-times spread in resolve rate between a baremodel call and a well-engineered scaffolded loop on the same checkpoint. That gap is system performance, not model performance.
Method claims require controlled ablations: same compute, same data, same architecture except for the proposed change. Reporting only the fully optimized system score is not sufficient to support a method claim.
When you cannot disentangle variables (e.g., multiple changes shipped simultaneously), report the con-founding explicitly. A confounded result is not useless-it tells you the system improved-but it does not let you allocate credit to a cause.

23.1.2 Ask-Style vs. Quiz-Style Prompts

An interviewer hands you a model that scores 87% on HumanEval. You are asked to decide whether to roll it out as the primary coding assistant for your engineering org. The number feels solid-but the question you need answered is not “can this model solve LeetCode problems someone has already verified?” The question is “will engineers accept its suggestions when they are stuck on real, ambiguous work at 2 am?” Those are fundamentally different prompts, and the gap between them is where most production deployments quietly underperform their benchmark numbers. This distinction-ask-style versus quiz-style-is one of the most practically important ideas in LLM evaluation, and it comes up whenever an interviewer wants to probe whether you understand why benchmark scores do not automatically transfer to product outcomes.

Every evaluation prompt can be classified by whether the person writing it already knows the correct answer.

A quiz-style prompt is one where the evaluator holds the ground truth before the model sees the question. Standardized benchmarks-MMLU, GPQA, HumanEval, GSM8k-are entirely quiz-style. A human expert (or an automated pipeline) constructed the question, verified the answer, and packaged both into a data set. The model’s job is to reproduce the known answer. The evaluator’s job is to check whether it did.

An ask-style prompt originates from a user who genuinely does not know the answer and is submitting the prompt because they need help. The user asking Claude to draft a differential diagnosis for a patient they are currently treating, or to explain why their Kubernetes pod keeps OOMing, has no verified ground truth in hand. They are trying to extract value from the model, not test it.

The distinction is not just philosophical. It determines what the score actually predicts. Quiz-style scores predict performance on other quiz-style tasks drawn from the same distribution. Ask-style scores-which require either human judges, LLM-as-judge pipelines, or downstream behavioral proxies like acceptance rate-predict whether real users get value from the system. The two correlate weakly at best once you move outside the model’s training distribution.

A benchmark score tells you how well a model performs on problems where someone already knows the answer; what matters in production is how well it performs on problems where nobody does.

Quiz-style benchmarks dominate the evaluation literature for one reason: they are cheap to construct and cheap to score. You do not need human judges. You do not need live users. You need a question, an answer key, and a string-matching or execution harness. That operational simplicity is why MMLU has 14,000 questions and GPQA has 448, and why you can reproduce either in a single script.

The cost is validity. Quiz-style prompts are clean, precisely specified, and have unambiguous single correct answers. Real user queries are underspecified-“fix my code” with no error message, “summarize this” with no stated audience. A model trained on internet text and evaluated on quiz-style prompts derived from the same internet can score well while failing badly on genuine user need.

Ask-style evaluation is harder operationally but more predictive. Analyses of real conversation traffic find coding is the dominant use case, followed by writing assistance and analysis. None of those sessions look like multiple-choice questions. The prompts are long, messy, contextual, and often span multiple turns. A benchmark built from that traffic-with humans or calibrated LLM judges scoring helpfulness rather than correctness-tracks production quality far better than any standardized exam.

The tension is that realistic ask-style data is often private. Clinical notes, customer service transcripts, and internal code reviews are the highest-value ask-style prompts, and they cannot be published. This is why initiatives like MedHELM exist: 29 clinicians identified 121 tasks reflecting their actual practice, and the resulting benchmark uses realistic prompts (patient note summarization, treatment plan drafting) that no multiple-choice exam covers. The privacy-versus-realism trade-off is real, and the right answer is almost always to invest in the ask-style eval even if it cannot be open-sourced.

A Worked Example

Suppose you are evaluating a code completion model for internal deployment. You run two evaluations in parallel.

Quiz-style (HumanEval). 164 Python function stubs with docstrings and unit tests. The model generates a completion; you execute the tests; you report pass@1. Your new checkpoint scores 87%, up from 80% on the previous release. The delta of 7 points looks meaningful.

Ask-style (sampled production sessions). You sample 2,000 real completions accepted or rejected by engineers over the past two weeks. A human rater panel (or a calibrated LLM judge) scores each completion on a 1-5 helpfulness scale, blind to which model generated it. You also instrument the IDE plugin to record whether the engineer accepted the suggestion, modified it substantially, or discarded it. Results: mean helpfulness score 3.4 vs. 3.1 for the prior release ( $+$ 0.3 points), and acceptance rate 54% vs. 51% ( $+$ 3 points). Both improvements are real but smaller in magnitude than the HumanEval delta suggests.

The gap exists because HumanEval problems have clean docstrings and self-contained scope. Real tickets say things like “add the same caching logic we use in UserService but for OrderService”-which requires reading a second file, understanding an undocumented convention, and matching a style the model has never seen. Ground truth requires a self-contained problem; real work is rarely that. The 7-point HumanEval gain measures something narrower than “will engineers be more productive?” The accept-rate gain of 3 points answers the question that actually matters.

What You Actually Decide in Practice

Use quiz-style benchmarks to track training progress, not to make deployment decisions. They are reproducible, fast, and cheap. They are the right tool for deciding whether a checkpoint is heading in the right direction during development. They are the wrong tool for deciding whether to ship it.
Build at least one ask-style eval before launch. The minimum viable version is a few hundred real prompts from your target user population, scored by human raters or an LLM judge calibrated against human raters on a held-out slice. If you cannot get real prompts pre-launch, use synthetic ones drawn from a model of your users’ tasks-but acknowledge the limitation.
Instrument for behavioral proxies when human rating is too slow. Acceptance rate, edit distance after acceptance, session abandonment rate, and thumbs-up/thumbs-down feedback are all ask-style signals that accumulate automatically once you are in production. Wire these up before launch so you have a baseline to compare against when the next checkpoint ships.
Never report only quiz-style scores for a safety-sensitive deployment. A model that scores 86% on USMLE Step 2 CK may still hallucinate drug dosages at a rate that is clinically unacceptable. Safety evals must be ask-style, grounded in the actual failure modes that harm real users, not the failure modes that fit neatly into a multiple-choice format.
Watch for quiz contamination in user feedback. Live traffic from a deployed model is not automatically ask-style. A significant fraction of real sessions are engineers or researchers actively probing the model’s limits-quiz-style behavior from real users. If your user population skews toward power users or developers, your “live traffic” evaluation may overestimate performance on genuine end-user need. Segment by user cohort before drawing conclusions.

How This Shows Up in Interviews

“Your model hits 90% on MMLU. How do you decide whether to deploy it to customers?” Strong answer: MMLU is a quiz-style benchmark and tells you the model can retrieve factual knowledge under a standardized prompt format. Before deployment you need an ask-style eval on a representative sample of your actual customer tasks-measuring helpfulness, groundedness, and refusal behavior-because MMLU accuracy does not predict those. Name a specific eval you would run (e.g., a human-judged sample of 100 realistic customer sessions) and what pass threshold you would require.
“We’re seeing a gap between our benchmark scores and user satisfaction in production. What’s going on?” Strong answer: the gap almost always traces to the quiz/ask mismatch. Benchmarks have clean ground truth; production prompts are underspecified and context-dependent. Diagnose by sampling failing production sessions and checking whether they share structural properties-long context, cross-file references, ambiguous intent-that are absent from the benchmark. Then build a targeted ask-style eval that captures those properties.
Senior follow-up: “How would you build an ask-style eval for a general-purpose assistant where there is no single correct answer?” Senior answer: layer three signals. Use LLM-as-judge at scale for triage (fast, cheap, biased toward length and verbosity). Audit a random 5% slice with human raters and report judge calibration-if the LLM judge agrees with humans 80% of the time you know how far to trust the automated scores. Track behavioral proxies (acceptance rate, task completion, return rate) as a continuous production signal. A score reported without calibration metadata is not an evaluation; it is an anecdote.

Key Takeaways

Quiz-style prompts have a known answer before the model sees the question; ask-style prompts originate from users who genuinely need help. The distinction determines what the resulting score predicts about production behavior.
Quiz-style benchmarks are cheap and reproducible-use them to guide training. Ask-style evals are expensive but valid-use them to make deployment decisions.
The quiz/ask gap is largest when real user traffic is underspecified, context-dependent, or drawn from a domain (clinical, legal, internal codebases) that is not well represented in public benchmarks.
Safety-critical deployments require ask-style evals grounded in realistic failure modes; a strong USMLE or MMLU score does not substitute for them.
A production ask-style eval needs a judging layer with measured reliability: report inter-rater agreement or LLM judge calibration alongside the helpfulness score, or the number is not trustworthy.

23.2 Metrics and Protocols

23.2.1 Perplexity as a Universal Signal

Every scaling law curve you have ever seen was fit against perplexity, not task accuracy. That is not an accident. When a senior engineer asks during an interview how you would track training health across a multi-week pre-training run, or how you would compare two architectural variants before committing to training one at full scale, perplexity is almost always the right first answer. Understanding both what perplexity measures and where it breaks down separates engineers who can read a training dashboard from engineers who can design the evaluation protocol behind it.

A language model defines a probability distribution $p$ over sequences of tokens. Given a held-out sequence $x_{1} , x_{2} , . . . , x_{N}$ , the model assigns a joint log-probability $\sum_{i =}^{N} \log_{1} p ( x_{i} | x_{< i} )$ . Perplexity is the exponential of the average per-token negative log-likelihood:

\mathrm{PPL} ( p , \mathcal{D} ) = \exp \Biggl( - \frac{1}{N} \sum_{i = 1}^{N} \log p ( x_{i} | x_{< i} ) \Biggr)

where $\mathcal{D}$ is a held-out data set and $N$ is the total token count. A lower perplexity means the model assigns higher probability to the observed text-it is less surprised. Perplexity of 1 is a perfect oracle; perplexity equal to the vocabulary size $| V |$ is a uniform random guesser.

Mechanically, perplexity during pre-training is a monotone transformation of the training objective. The gap between training loss and validation perplexity is the canonical sign of overfitting or distribution shift. Perplexity on a held-out slice of web text the model has never seen is the standard signal for tracking whether the model is learning to model language in general.

The perplexity-maximalist argument, which I find largely compelling for pre-training diagnostics, goes like this. If the true data-generating distribution is $t$ and your model is $p$ , then minimizing the KL divergence $D_{\mathrm{KL}} ( t \| p )$ is equivalent to minimizing cross-entropy, which is equivalent to minimizing perplexity. In the limit where $p = t$ , the model has matched human-level text generation and, in principle, can answer any question a human can answer. Perplexity, when computed against a broad and representative corpus, is measuring proximity to that ideal.

Perplexity is the only metric that assigns loss to every token uniformly, making it the most data-efficient signal for detecting whether a model is learning-and the right tool for fitting and extrapolating scaling laws.

Perplexity wins for pre-training diagnostics for three concrete reasons.

First, it is smooth. Task accuracy on a multiple-choice benchmark is a step function: you either produce the right letter or you do not. Perplexity is a continuous function of the model’s log-probabilities, which means you can detect a small improvement from a data mixture tweak or a learning-rate schedule change before that improvement is large enough to move a downstream accuracy number. This smoothness is why every published scaling law-Chinchilla, GPT-4’s training curves, Llama’s ablations-is expressed in terms of cross-entropy loss rather than MMLU scores. In the mid-2010s, the most striking result in language modeling was a paper from Google showing that a carefully scaled architecture could push Penn Treebank perplexity from roughly 51 down to 30. That 40% reduction signaled a sea change in what neural language models could do, long before downstream task benchmarks existed to measure it.

Second, perplexity is architecture-agnostic. If you are comparing two models that differ only in attention variant and you want to know which is learning more efficiently per FLOP, perplexity on the same validation set is an apples-to-apples comparison. Task accuracy would require both models to be trained long enough to generalize to the task format, which may not happen for small ablations early in a run.

Third, perplexity is hard to game through prompt engineering. Win rates and task accuracies can be inflated by choosing the right few-shot examples, the right chain-of-thought template, or the right decoding strategy. Perplexity does not depend on any of these; it requires direct access to the model’s logits over the held-out text, and as long as train and test splits are separate, it is not a quantity you can easily overfit to.

The main weakness of perplexity is that it is not always predictive of downstream task performance on any particular task in the short run. A model can achieve lower perplexity on web text by getting better at predicting Reddit discussions and financial news, without improving on formal mathematical reasoning. Over large scale differences, perplexity and task performance tend to correlate because stronger models are better at most things. Within a narrow training regime or for a specific domain, the correlation can be weak.

A Worked Example

Consider a pre-training ablation: you want to know whether increasing the proportion of code data from 15% to 25% of your training mixture helps a 7B model ( $d_{\mathrm{model}} = 4096$ , 32 layers, 32 heads, bf16, trained on 300B tokens across 64 H100s).

You train two ablation models, each for 10B tokens, on the two mixtures. After 10B tokens, Model A (15% code) has validation perplexity 8.4 on a held-out web text slice and 12.1 on a held-out code slice. Model B (25% code) has 8.6 on web text and 11.3 on code.

From this you conclude: the code-heavy mixture slightly hurts general web text modeling (perplexity rises by 0.2) but improves code modeling by 0.8 perplexity points. Had you used HumanEval pass@1 instead, you would need to run much longer before either model generates enough correct completions for a statistically stable comparison. Perplexity gives you the direction after 10B tokens; a task-accuracy signal would require 50-100B tokens before variance is low enough to trust.

You also watch per-token loss curves during training. If validation perplexity flattens while training loss keeps falling, you have a data contamination or overfitting signal. If perplexity spikes sharply on a particular batch, you have a data quality issue or a bad learning-rate spike. Neither of these would show up as a taskaccuracy alarm until much later.

What You Actually Decide in Practice

Use perplexity for all training-time decisions and ablations. Any decision made before the model completes pre-training-data mixture, learning rate schedule, architecture tweak-should be based on perplexity, not downstream task accuracy. Task accuracy is too noisy at small scale and too expensive to evaluate continuously during a run.
Match tokenizers when comparing perplexities across models. Perplexity is a function of the token sequence, not raw text. A model with a larger vocabulary will have lower perplexity on the same text because it segments tokens more coarsely. If tokenizers differ, normalize to bits-per-byte or bits-per-character before drawing any conclusions.
Require logit access before trusting a reported perplexity. If you are running a leaderboard that accepts model submissions, perplexity is easy to misreport. A model that assigns 0.8 to every next-token position-a non-distribution-will appear to have low perplexity but is meaningless. Verify that reported logits sum to 1 over the full vocabulary, or compute perplexity yourself from open-weight checkpoints.
Use domain-stratified perplexity, not a single aggregate. One number averages over all text types. Split your validation set into slices-web text, code, math, multilingual-and track perplexity per slice. This is what reveals whether a data mixture change helped one domain at the cost of another, which a single aggregate will hide.
Perplexity does not replace task evaluation at deployment time. Once you are making a deployment decision-should we ship this checkpoint?- perplexity is necessary but not sufficient. Two models with identical web text perplexity can differ by 10 points on HumanEval if one was trained with more code. Use perplexity to filter candidates during training; use task accuracy to select among finalists.

How This Shows Up in Interviews

“How would you track training health during a long pre-training run?” Strong answer: monitor validation perplexity on a domain-stratified held-out set, tracking both aggregate and per-domain curves. Set up automated alerts for perplexity spikes (indicating data or learning-rate issues) and for validation-training divergence (indicating overfitting or contamination). Task accuracy is too slow to provide the real-time feedback you need during training.
“A colleague says their new data mixture improved MMLU by 2 points after 10B training tokens. Is that a reliable signal?” Strong answer: almost certainly not. At 10B tokens, MMLU variance across seeds is typically larger than 2 points. The right signal at 10B tokens is perplexity on domain-stratified validation slices. Ask them to show the perplexity curves before treating the MMLU number as meaningful.
Senior follow-up: “When is perplexity insufficient as your only signal, even during pre-training?” Senior answer: perplexity is insufficient when the capability you care about requires compositional generalization rather than next-token prediction-for example, multi-step mathematical reasoning or long-horizon planning. A model can minimize perplexity on math corpora by learning to predict common surface patterns without being able to solve novel problems. For these capabilities you need targeted heldout task evaluations at training time-not as the primary optimization signal, but as a periodic diagnostic. Some training runs at frontier labs use “canary tasks”-small, carefully constructed task suites evaluated every $N$ steps-precisely to detect when perplexity progress stops translating to capability progress.

Key Takeaways

Perplexity is $\exp \bigl( -^{\frac{1}{N}} \sum_{i} \log p ( x_{i} | x_{< i} ) \bigr)$ , the exponential of average per-token cross-entropy. Lower is better; it measures how surprised the model is by held-out text.
Its smoothness and token-level granularity make it the right signal for training-time decisions: data mixture ablations, learning-rate tuning, and architecture comparisons at small scale.
Perplexity requires raw logit access and is easy to misreport. Require that reported logits sum to 1 over the full vocabulary, and normalize to bits-per-byte when comparing models with different tokenizers.
Use domain-stratified perplexity to detect trade-offs between text domains that a single aggregate number will hide.
Perplexity is necessary but not sufficient for deployment decisions. Use it to guide training; add taskaccuracy evals to select among final checkpoint candidates.

23.2.2 Exact Match, F1, Pass@k, and Win Rates

Once you move past perplexity and onto measuring what a model actually produces when deployed-answers to questions, code patches, explanations, clinical summaries- you have to choose a task-level metric. That choice is not cosmetic. I have seen teams report a 5-point accuracy gain that evaporated when they switched from exact match to F1, and I have seen code generation win rates swing 15 points when the evaluation moved from pass@1 to pass@10. The metric is a design decision with real consequences, and interviewers at any lab with a production system will probe whether you understand those consequences.

Four metric families cover the vast majority of LLM evaluation scenarios.

Exact match (EM) compares the model’s output, after normalization (strip punctuation, lowercase, collapse whitespace), to a reference string. It scores 1 if they are identical, 0 otherwise. This is the right metric for closed-form, unambiguous outputs: MMLU letter choices, arithmetic results, entity extraction with a canonical form.

Token-level F1 treats the prediction and reference as bags of tokens and computes precision and recall over the overlap:

F_{1} = \frac{2 \cdot | \text{pred} \cap \text{ref} |}{| \text{pred} | + | \text{ref} |}

This gives partial credit for approximately correct answers. It is the standard for open-domain QA tasks like SQuAD and TriviaQA, where a reference answer may have multiple valid phrasings and a word-for-word match is too strict.

Pass@k is used for code generation and any task where the model’s output can be automatically tested for correctness. Rather than literally drawing $k$ samples and checking, you draw $n \gg k$ samples and use the unbiased estimator:

\mathrm{Pass}@k=\mathbb{E}_{\text{problems}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]

where $n$ is the number of samples drawn per problem and $c$ is the number that pass all unit tests. The formula gives the probability that at least one of $k$ randomly selected samples from the $n$ drawn is correct.

Win rates are used when there is no ground-truth reference: open-ended generation, creative writing, instruction following, summarization. You present pairs of model outputs-to human judges or to a judge LLM-and record which output the judge prefers. Aggregating pairwise preferences via an ELO or Bradley-Terry model yields a scalar rating that is comparable across models. Chatbot Arena applies this protocol at scale with human judges; AlpacaEval applies it automatically with GPT-4 as judge.

The right metric is determined by the task’s output space: closed-form answers use EM; partial-credit spans use F1; verifiable programs use Pass@k; open-ended generation requires a judge, human or model.

Each metric trades precision against breadth of applicability in a different way.

Exact match is maximally precise but brittle. A reference of “Paris” and a prediction of “Paris, France” score 0 under EM despite being semantically equivalent. You pay for that brittleness with simplicity: EM is fast, requires no judge, and cannot be gamed once the answer form is fixed.

Token-level F1 buys robustness to surface variation at the cost of rewarding verbosity. A model that outputs a very long prediction containing the correct tokens will inflate its recall while precision stays low. This is why SQuAD-style QA systems often apply the min of prediction and reference length, or use a normalizer that strips articles and punctuation, to reduce the surface area for gaming.

Pass@k decouples evaluation from deployment scenario in a way that matters for production. Pass@1 measures the probability that a single sample is correct-relevant if you show users one completion. Pass@10 measures the probability that at least one of 10 samples is correct-relevant if your system generates multiple suggestions and picks the best via a reranker or test execution. The gap between pass@1 and pass@10 reveals how much diversity the model has: high pass@10 with low pass@1 means the model generates varied samples and benefits from sampling; nearly equal pass@1 and pass@10 means the model is low-entropy and additional samples add little value.

Win rates via LLM judges are convenient but systematically biased. GPT-4-as-judge prefers longer, more verbose outputs even when they are not more correct-a well-documented artifact that caused inflated win rates for models that learned to pad responses, until length-controlled variants of AlpacaEval corrected for it. On the human side, Chatbot Arena judges are representative of whoever visits the site, which may not match your target user population. A leaderboard that becomes a PR target also attracts gaming: the “Leaderboard Illusion” paper documented cases of providers gaining privileged submission access or cherry-picking model configurations, eroding the benchmark’s integrity.

A Worked Example

Consider evaluating a 7B code model (Llama-3-style, $d = 4096$ , 32 layers, bf16, context 8192 tokens) on HumanEval, which contains 164 programming problems with canonical unit test suites.

Draw $n = 20$ samples per problem at temperature $T = 0 . 8$ , top- $p = 0 . 95$ . Run the unit tests on each sample. For problem 42, suppose 8 of 20 samples pass, so $c = 8$ , $n = 20$ .

For $k = 1$ :

\mathrm{Pass}@1=1-\frac{\binom{20-8}{1}}{\binom{20}{1}}=1-\frac{12}{20}=0.40

For $k = 10$ :

\mathrm{Pass}@10=1-\frac{\binom{12}{10}}{\binom{20}{10}}=1-\frac{66}{184{,}756}\approx0.9996

The ratio pass@10 / pass@1 $\approx 2 . 5$ for this problem indicates that the model knows how to solve it but has significant output variance-additional samples are worth generating if the deployment context allows it. Averaged over all 164 problems, a 7B model at this training scale typically achieves pass@1 $\approx 0 . 55$ - $0 . 65$ and pass@10 $\approx 0 . 75$ - $0 . 85$ , depending on the training mixture. That gap is not noise; it reflects a genuine difference between showing one suggestion versus ten.

Now suppose you also run a win-rate evaluation on the same model’s docstring generation capability. You generate 500 docstrings for real functions from your codebase, pair each against the previous checkpoint’s output, and ask GPT-4o to judge which is more accurate and concise. You observe a raw win rate of 53%. When you filter to responses under 100 tokens-removing the verbosity confound-the win rate drops to 50%. The raw 53% was length bias, not quality improvement. Reporting only the raw number would have led to a false conclusion about the new checkpoint.

What You Actually Decide in Practice

Default to pass@1 for user-facing deployments, pass@k for capability research. When reporting what users actually experience with a single-suggestion system, pass@1 is the number that matters. Use pass@k to characterize ceiling capability or to evaluate systems that explicitly generate and rerank multiple candidates.
Use the unbiased pass@k estimator, not a hard threshold. Computing pass@k by generating exactly $k$ samples and checking is high-variance for small $k$ . Generate $n \geq 20$ samples per problem and use the $\binom{n - c}{k} / \binom{n}{k}$ estimator. This is what the original HumanEval paper recommends and what you should reproduce for comparability.
Length-correct your win rates before reporting them. Any LLM-as-judge win rate should be accompanied by a length-controlled variant where output length is accounted for. If the length-corrected win rate differs from the raw win rate by more than 2 absolute points, report both and investigate before attributing the gain to quality.
Pair every safety metric with a capability metric. A model that refuses everything scores 100% on harm benchmarks. Safety numbers are uninterpretable without a simultaneous report of pass@1 on coding, exact match on factual QA, or some other capability signal. Report them together, not in isolation.
Put cost on the Pareto frontier, not in the denominator. A model with pass@1 $= 0 . 70$ that costs 10 $\times$ more per token than one with pass@1 $= 0 . 68$ is not obviously better for production. Report accuracy at fixed latency budget or accuracy per dollar when comparing models with different serving costs. Many public leaderboards omit cost entirely; supplement them with a Pareto frontier if you are making a deployment decision.

How This Shows Up in Interviews

“How would you evaluate a code generation model for production deployment?” Strong answer: use pass@1 at the system’s actual sampling temperature as the primary accuracy metric; supplement with pass@10 to characterize ceiling capability; measure latency-to-first-token and cost per accepted suggestion; report accuracy vs. cost on a Pareto frontier rather than a single number.
“A team reports a 5-point win rate improvement on AlpacaEval. What should you check before trusting that number?” Strong answer: check whether the improvement survives length correction; check whether the same improvement appears on Chatbot Arena or a human evaluation on your actual task distribution; verify the baseline was evaluated under identical decoding parameters (temperature, top- $p$ , maximum tokens). A win rate that disappears after length correction is measuring verbosity, not quality.
Senior follow-up: “Your pass@1 improved by 4 points on HumanEval, but users are not reporting a better experience. What might explain the gap?” Senior answer: HumanEval problems are short, self-contained, and have canonical test suites. Production code generation tasks involve existing codebases with implicit context and rarely have test suites. A 4-point HumanEval gain can come from better performance on easy function-completion problems that are already mostly solved in production. Run an evaluation on a held-out sample of real user sessions-accept rate, or user edit distance after accepting a suggestion-to determine whether the benchmark gain translates. If it does not, the gap is in the evaluation distribution, not the model, and you should redesign the benchmark before acting on future HumanEval results.

Key Takeaways

Exact match is precise but brittle-use it only for closed-form, canonical outputs such as classification labels and arithmetic results.
Token-level F1 gives partial credit for span-extraction tasks; watch for verbosity inflation and normalize output length if the model’s responses are unconstrained.
Pass@k decouples capability from deployment scenario. Use pass@1 for single-suggestion systems, pass@k for best-of- $N$ or capability research, and always apply the unbiased estimator with $n \geq 20$ samples per problem.
Win rates require a judge and are systematically biased toward longer outputs. Always report a lengthcontrolled variant alongside the raw win rate.
Pair every accuracy or safety metric with a cost metric. Accuracy at fixed latency budget or accuracy per dollar is the right unit for deployment decisions.