Chapter 25 Validity, Contamination, and Real-World Use

25.1 Train-Test Overlap and Contamination

25.1.1 Detecting Contamination

Any time you run a model on a benchmark that predates its training cutoff, the reported score conflates two signals: genuine generalization and memorization of test examples that leaked into training data. At senior-level research and systems interviews, interviewers expect you to distinguish these signals and to describe evaluation pipelines that survive the challenge. The contamination problem was tractable when training sets were curated splits of fixed corpora; it became nearly unmanageable once web-scale crawls became the default, because no one can fully audit what three trillion tokens of web text contains.

Train-test contamination occurs when one or more test examples - or near-duplicate paraphrases of them - appear in the model’s training data. The model then solves the benchmark partly by recall rather than by the generalization the benchmark was designed to measure.

In the pre-internet era this was straightforward to avoid: a benchmark designer partitioned a fixed corpus into designated train and test splits, and everyone trained exclusively on the training partition. The model developer had explicit custody of their data and could prove the split was clean.

Today the problem is structurally different. A model trained on a web crawl ingests pages that themselves quote benchmarks. Test preparation sites reproduce MMLU questions verbatim. Math tutoring sites contain GSM8K problems. Code repositories contain HumanEval examples. None of this was placed there deliberately, but it ends up in pre-training data anyway. Worse, the developer often cannot prove the contrary, because the training corpus was never fully audited at example level.

The standard mitigation is n-gram decontamination: build a fingerprint set from all test examples, scan the training corpus for matching n-grams, and remove any document that contains a match. Thirteen-gram overlap is the most common choice. For a test example $t$ of $L$ tokens, the fingerprint set is:

F ( t ) = \bigl\{ ( w_{i} , w_{i + 1} , . . . , w_{i + 12} ) \bigr| \bigr| 1 \leq i \leq L - 12 \bigr\}

A training document $d$ is flagged and removed if $F ( t ) \cap \mathrm{ngrams}_{13} ( d ) \neq \emptyset$ for any benchmark test example $t$ .

N-gram decontamination catches verbatim and near-verbatim leakage, but is blind to semantic overlap that preserves the information content of a test example while evading string matching entirely.

The 13-gram threshold sits at a deliberate point in a precision-recall trade-off. Shorter n-grams (say, 5- gram) produce enormous false-positive rates: common phrases like “what is the correct answer” appear everywhere and would remove vast swaths of legitimate training data. Longer n-grams (say, 20-gram) reduce false positives but require a near-verbatim match, missing documents that paraphrase the question with minor edits.

The fundamental limitation is that n-gram decontamination is a string-matching procedure operating on a semantic problem. Three failure modes matter in practice:

Near-duplicates below the threshold. A document that reproduces a test question with a single substituted word will share fewer than 13 consecutive tokens with the fingerprint and will not be flagged. For short questions (under 20 tokens), even a two-word change evades detection entirely.
Cross-lingual paraphrases. A math problem translated into another language has zero token-level n-gram overlap with the English original. A model capable of cross-lingual reasoning can still solve the translated version by recall if the translated problem appeared in multilingual training data.
Indirect quotation. A blog post that discusses a benchmark question (“here is an example MMLU question, and here is why the answer is C”) may rephrase rather than reproduce the original, yet still teach the model the correct answer.

For black-box models where the developer does not disclose training data, external contamination detection relies on behavioral probes. The most reliable is an ordering probe: shuffle the answer choices of multiple-choice questions and measure whether the model assigns systematically higher probability to orderings that match the original benchmark ordering. A model that has memorized the test set retains the specific answer-choice order; one that has genuinely generalized does not care. A second signal is confidence asymmetry: compare model confidence on held-out examples that are semantically equivalent to test examples but were constructed after the training cutoff; a large gap in confidence between the two sets is evidence of memorization.

A Worked Example

Consider decontaminating a 3-trillion-token training corpus against the MMLU test set (14,163 questions, each averaging roughly 80 tokens including answer choices).

Tokenize all test examples. Compute all 13-grams: approximately $14 , 163 \times ( 80 - 12 ) = 963 , 000$ raw fingerprints; after deduplication, around 800,000 unique 13-grams.
Store the fingerprint set in a hash set. At 26 bytes per 13-gram of 16-bit token IDs, this occupies roughly 21 MB - trivially small.
Scan the training corpus document by document with a sliding 13-gram window. For a 500-token document, generate 488 candidate 13-grams and probe the hash set for each.
Flag and remove any document with at least one hit. In practice, this removes around 0.05-0.2% of documents from a web-scale corpus - a small absolute loss.

What this misses: suppose a test prep site rephrases an MMLU chemistry question by substituting “for-mulated” for “developed” and “proposed” for “introduced.” After substitution, the longest shared run of tokens might be 8 consecutive tokens - below the 13-gram threshold. The document is not flagged, yet the model can observe the correct answer in training. The conservative response is to also run fuzzy deduplication (MinHash LSH at Jaccard similarity $\geq 0 . 5$ ) on the full question text, catching near-duplicates that evade exact n-gram filtering at roughly 3 $\times$ the computational cost of exact matching.

What You Actually Decide in Practice

Choose 13-gram as the default, but add fuzzy matching for high-stakes evals. Exact 13-gram decontamination is cheap and catches the most common leakage vectors. For benchmarks where every percentage point matters, add LSH-based near-duplicate detection as a second pass.
Decontaminate at document level, not passage level. Removing only the contaminated passage while keeping the surrounding document leaves context that still signals the answer. When in doubt, remove the whole document.
Be conservative: the cost of a false positive is low, the cost of a false negative is high. If your training set has 3 trillion tokens, removing 0.2% costs you 6 billion tokens - a rounding error. Failing to remove a contaminated document can inflate a score by multiple percentage points on a 14K-question benchmark.
Cross-lingual contamination requires semantic matching, not n-gram matching. For multilingual corpora or benchmarks with translated versions, run embedding-based similarity (e.g., cosine similarity of multilingual sentence embeddings above 0.9) as an additional filter.
For black-box models, run ordering probes before citing their benchmark numbers. Prepare a version of the benchmark with answer choices in reversed order. A score gap larger than 2-3 percentage points between original and shuffled-order versions is a red flag for memorization.

How This Shows Up in Interviews

“A model you trained gets 91% on MMLU. How do you know it actually learned something and didn’t just see the test set?” A strong answer: run 13-gram decontamination, verify the removal rate was nonzero, and run an ordering probe on the evaluated model. If the score drops significantly under shuffled-choice ordering, contamination is likely.
“What is n-gram decontamination and when does it fail?” The strong answer names the three failure modes: near-duplicates below the threshold, cross-lingual paraphrases, and indirect quotation. It also states the standard threshold (13-gram) and the correct default behavior (document-level removal).
“You are evaluating a closed-weight model and the developer claims the test set was not in training data. You have no access to training data. How do you verify?” This is the senior question. The strong answer describes two behavioral probes: an ordering probe (does the model’s preference for specific answerchoice orderings correlate with the original benchmark ordering?) and a confidence asymmetry test (does the model assign systematically higher log-probability to test examples than to semantically equivalent paraphrases constructed after the training cutoff?). Neither is conclusive alone, but both together give strong signal.

Key Takeaways

Train-test contamination is structural in web-scale training: test prep sites, tutoring resources, and repository files routinely reproduce benchmark examples verbatim.
Standard mitigation is 13-gram decontamination with document-level removal; adding LSH-based fuzzy matching catches near-duplicates that evade exact matching.
N-gram filtering is blind to semantic overlap, including cross-lingual paraphrases and indirect quotation - these require embedding-based detection.
For black-box models, behavioral probes - ordering probes and confidence asymmetry tests - provide contamination signals without access to training data.
When in doubt, remove more rather than less: losing 0.2% of web-scale training data is a trivial cost compared to reporting an inflated benchmark score.

25.1.2 Reporting Norms and Reproducibility

A benchmark score reported without its context is nearly uninterpretable. When an interviewer asks you how to evaluate whether a 91% MMLU score is meaningful, the first question is not “how did the model perform?” - it is “what was the prompt format, what were the decoding settings, and was the test set decontaminated?” These questions reveal whether the number measures the model or measures the evaluation setup. At both IC and staff-level interviews, this class of question tests whether you understand evaluation as an infrastructure problem, not just a statistical one.

Reproducibility in benchmark evaluation means that an independent team, starting from the same model, can arrive at the same score. This sounds obvious, but in practice it fails along two independent axes: protocol variance and data quality.

Protocol variance arises because the same model evaluated under different prompting strategies, few-shot example counts, or decoding settings can produce meaningfully different scores. On MMLU, the difference between zero-shot and five-shot evaluation can exceed 5 percentage points for mid-size models. A score reported without specifying these parameters is ambiguous by construction. Reproducible evaluation requires locking and disclosing at minimum: the exact prompt template, the number and source of few-shot examples, the sampling temperature and top- $p$ , and the model version or checkpoint hash.

Data quality failures occur when the benchmark’s reference answers are themselves wrong. Label noise is more common than the field acknowledges. When scores on a benchmark approach ceiling - say, above 85% - a significant fraction of the remaining “errors” may be the model answering correctly against a wrong reference label rather than genuinely failing. This makes it impossible to distinguish a 90% score that reflects true capability from a 90% score that reflects 95% true capability combined with 5% mislabeled examples.

A third, often overlooked norm is decontamination disclosure: whether the evaluator checked that the test set was not in the training corpus, and if so, by what method. The analogy to classical statistics is reporting a standard error alongside a point estimate. Neither the point estimate alone nor the standard deviation alone is sufficient to interpret the result; the same is true of a benchmark score and its contamination status.

A benchmark score without a protocol description, a contamination attestation, and a data quality audit is a number in search of a meaning.

Standardized evaluation frameworks such as lm-evaluation-harness and HELM address protocol variance by fixing the prompt template, few-shot selection logic, and decoding configuration as code that anyone can inspect and rerun. The reproducibility gain is significant: for any benchmark hosted in such a framework, the “evaluation protocol” is pinned to a specific commit hash and is fully auditable.

The trade-off is that standardized frameworks freeze a particular evaluation protocol. If the frozen protocol is suboptimal - for example, it uses a verbose few-shot preamble that handicaps instruction-tuned models - then all scores under that framework carry a systematic bias that is only visible if you re-run under an alternative protocol. The right response is not to distrust frameworks, but to always report the framework version and to flag when a significant deviation from the standard protocol was used.

Label noise poses a subtler threat because it interacts with capability improvements. As models improve, they are increasingly likely to produce the correct answer on questions where the reference label is wrong. This causes scores to plateau or even decrease near the benchmark’s ceiling, because the model is being penalized for answers that are correct. Fixing label errors in a benchmark after it has been widely cited creates a discontinuity: scores reported before and after the fix are not directly comparable.

A Worked Example

Consider the GSM8K math benchmark, which has 1,319 test problems. Suppose analysis reveals that 4% of reference answers contain errors (approximately 53 problems). A model achieving 90% measured accuracy ( $1 , 187 / 1 , 319$ correct) might actually be correct on a different set of problems than the leaderboard implies.

Define: $N = 1 , 319$ total, $E = 53$ mislabeled, $N^{\prime} = 1 , 266$ clean. If the model correctly answers all clean problems and none of the mislabeled ones:

\text{measured accuracy} = \frac{1 , 266}{1 , 319} \approx 96 . 0 \%

Conversely, if the model has genuine errors on 132 clean problems but happens to match all 53 wrong reference labels:

\text{measured accuracy} = \frac{( 1 , 266 - 132 ) + 53}{1 , 319} = \frac{1 , 187}{1 , 319} \approx 90 . 0 \%

Both scenarios produce 90%, yet one reflects 94.6% true accuracy on clean examples and the other reflects 89.5%. Without a data quality audit, these are indistinguishable from the score alone. The operational consequence is clear: when benchmark scores reach the upper eighties and above, always audit a sample of “errors” before concluding the model has hit a capability ceiling.

What You Actually Decide in Practice

Pin three things before reporting any score: framework version, prompt template, and few-shot count. A score reported with “we used lm-eval-harness v0.4.2, zero-shot, default template” is reproducible. “We evaluated on MMLU” is not.
Require decontamination disclosure as a reporting norm, not an optional footnote. When you publish evaluation results, state explicitly: (a) what decontamination method was used, (b) the removal rate (percentage of training documents flagged), and (c) whether the post-decontamination score differed from the pre-decontamination score. A removal rate of 0% is informative: it tells the reader the method found nothing, which is different from not running the method at all.
Audit benchmark label quality before using it as a decision signal. For any benchmark you intend to use to make a deployment decision, randomly sample 100 examples and verify the reference answers manually. For benchmarks near ceiling, sample from the “errors” the model makes - if you find mislabeled items, the effective ceiling of the benchmark is below 100%.
Track score time-series, not point estimates. A model that improves from 85% to 87% on a benchmark is a signal; a model that jumps from 87% to 92% in one version update warrants investigation. Sudden large jumps more often reflect a change in evaluation protocol or a contamination event than a genuine capability leap.
For closed-weight models evaluated externally, treat scores as approximate ordinals, not precise quantities. A closed model can silently update between when you run an eval and when your paper is published. Reporting the model version, the API endpoint slug, and the date of evaluation gives future readers enough context to detect silent updates.

How This Shows Up in Interviews

“A colleague says our new model got 91% on MMLU. What follow-up questions do you ask?” A strong answer covers: which prompt format and few-shot count, which framework version, whether decontamination was run and what the removal rate was, and whether this is a base model or instruction-tuned model (since the same benchmark can behave differently for each). A weak answer says “that sounds good” or asks only “what was last version’s score?”
“What does it mean for a benchmark to be reproducible, and what breaks it?” Protocol variance (prompt format, decoding settings), data quality (label noise), and contamination status. Reproducibility requires locking all three. The most actionable fix is adopting a versioned evaluation framework that pins the protocol as code.
“A benchmark you care about shows model scores plateauing near 90% despite clear improvements on harder tasks. What is your diagnosis?” This is the senior question. The strong answer distinguishes three competing explanations: (1) genuine capability ceiling on the benchmark’s difficulty distribution, (2) label noise floor - the model can no longer improve beyond the noise rate, and (3) distribution shift between the model’s strengths and the benchmark’s coverage. The senior move is to audit 100 randomly sampled “errors” to discriminate between explanations 1 and 2 before concluding the model has hit a real ceiling.

Key Takeaways

A benchmark score is only reproducible when the prompt format, few-shot count, decoding settings, framework version, and decontamination protocol are all disclosed alongside it.
Label noise inflates apparent model accuracy near benchmark ceilings and creates ambiguity that cannot be resolved from the score alone; auditing a sample of model errors is necessary.
Decontamination disclosure should be a community norm analogous to reporting confidence intervals: state the method, the removal rate, and whether post-decontamination scores changed.
Standardized frameworks (lm-evaluation-harness, HELM) address protocol variance by pinning evaluation as versioned code; always report the framework version when citing scores.
Sudden large jumps in benchmark scores warrant investigation before being attributed to capability improvement - they more often signal a protocol change or a contamination event.

25.2 Real-World Evaluation

25.2.1 Deployment Telemetry and Live Traffic

When a model passes every curated benchmark on your eval rack and then underperforms in production, the problem is almost always distribution mismatch. Standardized benchmarks quiz models-the question designer already knows the answer and the prompt is crafted to probe a specific capability. Real users ask-they do not know the answer, the phrasing is messy, the intent is sometimes ambiguous, and they care only about whether the response actually helps them. Closing this gap is a core challenge for senior ML engineers building production systems, and interviewers at frontier labs ask about it directly when evaluating candidates for deployment or evaluation engineering roles.

Deployment telemetry treats live inference logs as a first-class evaluation signal. After a model is in production, every request is a data point: what prompt arrived, what the model returned, how long it took, whether the user continued the conversation, whether they copied the output, whether they hit the feedback button, and-if you instrument it-whether the downstream task succeeded. Aggregating these signals gives you a distribution of actual usage that no benchmark author could have anticipated.

The distinction between quizzing and asking is not just philosophical-it determines which metric you should trust. A quizzing prompt (MMLU, GPQA, AMC) has a ground-truth answer baked in; you can score automatically and the number is clean. An asking prompt (“help me draft an email declining this meeting”, “explain why my Kubernetes pod keeps OOMKilling”) has no canonical answer; the only reliable judge is whether the person who asked got what they needed.

Operationally, live-traffic evaluation takes two forms. Aggregate telemetry clusters conversations by topic and intent, then monitors quality proxies per cluster over time. Shadow evaluation runs a candidate model in parallel on the same traffic, compares outputs to the production model using an LLM judge or human raters, and reports a win-rate before the candidate is promoted.

Once you have paying or regular users, you have the best evaluation data in the world-use it.

Benchmark prompts, however carefully written, compress the tail of possible user behavior almost to zero. Production traffic is fat-tailed: the median prompt is unremarkable, but the 99th-percentile prompt is something the eval team never imagined. If your deployment serves millions of daily requests, the tail is enormous, and it is precisely the tail that reveals model failures-hallucinations on uncommon domain terms, refusals on legitimate edge-case requests, format breakdowns when the user pastes in a long document.

The alternative-relying solely on periodic offline evals-fails for two reasons. First, you discover regressions after users have already experienced them. Second, the offline distribution drifts from production as user behavior evolves; a new feature, a new user cohort, or even a seasonal shift in topics can invalidate a carefully tuned eval suite within weeks.

The trade-off is noise. Live traffic contains spam, prompt injection attempts, benchmark probers, and developers stress-testing the API. These dilute your quality signal. A raw thumbs-up rate across all traffic is almost meaningless without stratification by intent cluster and user type.

A Worked Example

Anthropic published an analysis of real Claude usage by running their own model over a large sample of production conversations and using it to assign soft cluster labels-essentially a scalable version of human topic coding. Coding tasks were the single largest cluster, followed by writing and summarization. This kind of analysis immediately tells you where to concentrate your evaluation budget: if 30% of your traffic is code generation and your offline eval suite has 5% code tasks, you are under-measuring your most important use case.

To make this concrete: suppose you pull one week of production logs, 10 million turns. You embed each user turn and cluster into $K = 200$ topics using $k$ -means over sentence embeddings. You then sample 50 turns per cluster and score them with a judge model on a 1-5 scale. You now have a per-cluster quality heatmap. You find that cluster 47 (“debugging Python asyncio code”) scores 2.1 on average while the overall mean is 3.8. That is your next fine-tuning target-not whatever the next benchmark paper happens to benchmark.

Shadow evaluation works like a canary deployment. The new model handles, say, 1% of traffic. A judge model scores both the shadow response and the production response on the same prompt. After 100k comparisons, you have a win rate with a tight confidence interval. A win rate above 55% is a meaningful signal to promote; below 50% means do not ship.

What You Actually Decide in Practice

Instrument before you launch. Logging is cheap; retroactive reconstruction of what happened is expensive. At minimum log: timestamp, prompt hash, model version, latency, output token count, and any explicit user feedback. Add conversation-level signals (session abandonment, follow-up clarification rate) as soon as feasible.
Stratify before you trust any aggregate metric. A single overall thumbs-up rate mixes spam with genuine queries, short factual lookups with 2,000-token coding tasks. Stratify by intent cluster, prompt length bucket, and user cohort before drawing conclusions.
Use shadow mode for every major model swap. Promoting a new model based solely on offline evals without a shadow run first is how you ship regressions. Shadow for at least 24 hours to capture diurnal patterns in traffic distribution.
Cap your spam fraction. If more than 20-30% of your logged traffic is probers, red-teamers, or clearly synthetic test traffic, filter it before building your quality heatmap. Leaving it in will make the model look worse on legitimate queries and better on pathological ones.
Budget for human annotation on the tail. LLM judges are fast and cheap but systematically miss certain failure modes-subtle factual errors, culturally inappropriate outputs, responses that are technically correct but practically useless. Reserve 2-5% of your annotation budget for human review on low-scoring cluster tails.

How This Shows Up in Interviews

“How would you evaluate a model once it’s in production?” A strong answer describes the two-track approach: aggregate telemetry for ongoing health monitoring and shadow evaluation for model upgrades. Mention stratification by intent cluster and the difference between proxy metrics (thumbs-up, session continuation) and direct quality scores.
“What’s wrong with using only Chatbot Arena or MMLU as your evaluation?” Point out that Chatbot Arena users are self-selected (whoever visits the site, which skews toward tech-savvy users testing the model, not using it), and MMLU is a quizzing distribution that does not resemble any deployed use case. Both are useful leading indicators but not substitutes for production telemetry.
Senior follow-up: “How do you detect a regression in production before users complain?” The senior answer involves anomaly detection on your quality heatmap: track the rolling mean score per cluster with exponential smoothing, alert when any cluster drops more than 0.3 points in a 6-hour window. Pair with latency and refusal-rate monitoring since safety over-refusals often accompany model regressions in the same rollout.

Key Takeaways

Benchmark prompts quiz models; real users ask models-these are different distributions and require different evaluation strategies.
Deployment telemetry is the highest-fidelity eval signal available once a model is live; build logging infrastructure before launch, not after.
Shadow evaluation-running a candidate model on 1-5% of live traffic and scoring against the production model-is the standard practice for safe model promotion.
Stratify all aggregate metrics by intent cluster and user type before drawing conclusions; raw averages over heterogeneous traffic are nearly meaningless.
Human annotation on cluster tails remains irreplaceable for catching failure modes that LLM judges systematically miss.

25.2.2 Domain-Specific Evaluations (MedHELM-Style)

General-purpose benchmarks tell you roughly how capable a model is. They tell you almost nothing about whether it is safe and useful for a specific high-stakes domain. A model scoring 90% on MMLU and 75% on GPQA may still hallucinate drug interactions, misread imaging reports, or produce legally incoherent contract clauses-because none of those benchmarks were built from practitioner workflows. Senior engineers evaluating models for specialized deployment are expected to understand how to move from generic leaderboard numbers to domain-grounded assessments, and how to navigate the privacy and annotation costs that come with that territory.

Domain-specific evaluation starts from a different question than general benchmarks: not “what can this model do on a curated test set?” but “which tasks does this domain actually require, and how well does the model perform on those?” The design process inverts the usual flow. Instead of researchers proposing tasks and then recruiting subjects, you recruit domain practitioners first, extract their workflows and pain points, and derive evaluation tasks from those.

MedHELM, a medical evaluation suite built on top of the HELM framework, is the clearest published example of this methodology. The authors recruited 29 clinicians across specialties and asked a simple question: where in your daily practice would a language model be useful? They identified 121 distinct clinical tasks. These were not exam questions. They included things like synthesizing a patient’s history into a structured note, drafting a referral letter, proposing a differential diagnosis given a case summary, and answering a patient’s follow-up question in plain language. Those tasks were then operationalized into concrete evaluation instancesprompts paired with rubrics or reference outputs that clinicians themselves validated.

The resulting benchmark has a fundamentally different character from MMLU. It is narrower in domain but far deeper in ecological validity: the tasks look like what the system will actually be asked to do in deployment.

The right evaluation asks the same questions the deployment will ask; anything else is measuring a proxy.

The failure mode of generic medical benchmarks is well-documented. For years, the standard way to evaluate a model on medicine was to run it on USMLE Step 1 and Step 2 questions-the licensing exams for American physicians. A model that aces USMLE has demonstrated that it memorized a large amount of structured medical knowledge. It has not demonstrated that it can produce a usable clinical note, counsel a patient about medication risks in understandable language, or flag when a case summary contains internally inconsistent lab values. Those are the tasks that create or destroy value in practice.

Practitioner-derived benchmarks close this gap by construction: the tasks were sourced from people whose job depends on executing them correctly. They also reveal capability distributions that generic evals obscure. A model may score 88% on a general medical knowledge benchmark but 52% on patient-facing communication tasks and 71% on clinical documentation-a profile that directly informs deployment decisions (use it for documentation, monitor carefully for patient-facing use, do not deploy autonomously for either).

The main trade-off is cost and access. Designing a practitioner-sourced benchmark requires significant expert time, which is expensive in any high-skill domain. And once you have real clinical cases-the highest-fidelity evaluation material-you immediately run into privacy constraints that prevent public release. This means the community cannot independently verify scores, and model developers must run the evaluation under controlled access agreements.

A Worked Example

Take the clinical note synthesis task from MedHELM. A clinician sees a patient with chest pain. After the visit, they need to produce a structured SOAP note (Subjective, Objective, Assessment, Plan) for the electronic health record. The evaluation instance looks like this:

Prompt:
  Patient: 58-year-old male, presenting with 2-day history of
  exertional chest pain, worse on inspiration. PMH: hypertension,
  type 2 diabetes. Current meds: metformin 1000mg BID, lisinopril
  10mg QD. Vitals: BP 148/92, HR 88, SpO2 97% on room air.
  ECG: sinus rhythm, no ST changes. Troponin x2 negative.

  Write a SOAP note suitable for the EHR.

Reference (clinician-authored):
  S: Patient reports 2-day exertional chest pain, pleuritic in
     character. Denies diaphoresis, radiation, or rest pain.
  O: BP 148/92, HR 88, SpO2 97%. ECG NSR, no ischemic changes.
     Troponin x2 negative.
  A: Chest pain, likely musculoskeletal vs. pleuritis. Low
     probability ACS given negative workup.
  P: Ibuprofen 400mg TID x5d, follow-up in 1 week, return
     precautions for worsening symptoms or rest pain.

The model output is scored on four axes by a clinician rater or a validated rubric: (1) factual accuracy-does it reflect what was in the prompt without adding hallucinated details; (2) structural correctness-does it follow SOAP format; (3) clinical appropriateness-is the assessment and plan medically sound; (4) usability-would the clinician accept this note with minimal edits. A model that scores well on a generic medical benchmark but produces a note attributing atrial fibrillation to a patient whose ECG explicitly shows sinus rhythm has a factual accuracy failure that the benchmark exposes immediately.

Across 121 tasks scored this way, you get a capability profile, not a single number. That profile is what an engineering team uses to scope deployment.

What You Actually Decide in Practice

Source tasks from practitioners, not from existing literature. Literature-based tasks inherit the same exam-question bias as standardized benchmarks. Ask clinicians, lawyers, or engineers what they do on a Tuesday, then build evaluation tasks around that.
Separate public and private splits deliberately. Release a subset of tasks publicly for reproducibility and community benchmarking. Keep the high-fidelity split (real patient records, real legal documents) private and run it under controlled access. The public split maintains accountability; the private split preserves validity.
Plan for multi-axis scoring from the start. A single accuracy score collapses the capability profile that practitioners actually care about. Budget for rubric design and, ideally, clinician review of at least a sample of model outputs. LLM judges can handle scale but require calibration against human scores in the specific domain.
Realism and privacy are in direct tension-resolve it explicitly. The most realistic evaluation uses real data. Real data often cannot be shared. Make this trade-off explicitly in your evaluation design documentation so downstream users understand what they are and are not measuring.
Expect 3-6 months of practitioner engagement to build the first version. Domain-specific benchmark construction is not a weekend project. Factor expert interview time, task validation cycles, and rubric calibration into your project plan before committing to a launch date.

How This Shows Up in Interviews

“How would you evaluate a model for deployment in a medical or legal context?” A strong answer describes the practitioner-interview process, contrasts it with standardized exam benchmarks, identifies the specific task types (documentation, summarization, patient communication), names the axes of evaluation beyond simple accuracy, and addresses the privacy constraint on using real data.
“Why can’t we just use USMLE or bar exam scores to evaluate domain models?” Point out the quizzing-versus-asking distinction: licensing exams test whether you have memorized structured knowledge under exam conditions, not whether you can execute the actual workflows of the profession. A model could pass Step 2 CK and still fail to write a usable discharge summary.
Senior follow-up: “How do you validate that your domain-specific benchmark actually predicts deployment quality?” The senior answer involves a calibration study: deploy the model on a small pilot with real practitioners, collect their satisfaction ratings and error reports, then measure the correlation between your benchmark scores and those outcomes. If a model that scores 0.4 on your rubric is flagged by clinicians 3 $\times$ more often than one that scores 0.7, your benchmark has predictive validity. If the correlation is weak, you need to revise the task set or the scoring rubric.

Key Takeaways

Generic benchmarks measure capability breadth; domain-specific evaluations measure deployment fitness-both are necessary, neither is sufficient alone.
Practitioner-interview-driven task design (as in MedHELM) produces evaluations with far higher ecological validity than repurposing existing standardized exams.
Multi-axis scoring (factual accuracy, format compliance, clinical or legal appropriateness, usability) gives engineering teams actionable deployment guidance; single-number scores do not.
Realism and privacy are structurally in tension in high-stakes domains: plan a public/private split from the start to balance reproducibility and validity.
Benchmark construction in specialized domains requires months of expert engagement; build this into project timelines before any model evaluation commitments are made.