Chapter 20 Pre-Training Data Pipelines

20.1 Where Training Data Comes From

20.1.1 Common Crawl, WET vs. WARC, and Extraction Tools

When an interviewer asks “where does your pre-training data come from?”, the honest answer for most large models is: mostly Common Crawl, with a lot of careful processing on top of it. Common Crawl is the closest thing the open world has to a snapshot of the web, and understanding its internals-specifically the difference between the WARC and WET formats, and what extraction tool you apply to convert HTML to text-is the kind of detail that separates an engineer who has built a real data pipeline from one who has only read about it. This topic shows up in senior ML engineer and research scientist interviews whenever the conversation gets into data quality or pipeline design.

Common Crawl is a nonprofit that has been running monthly web crawls since 2007. Each crawl visits billions of URLs using a seeded breadth-first traversal of the web, starting from hundreds of millions of seed URLs and respecting robots.txt directives. The result of each crawl is stored in two primary formats:

• WARC (Web ARChive): the raw HTTP response, including headers and the original HTML. This is the ground truth and can be tens of terabytes per snapshot.

• WET: a pre-extracted plaintext version produced by Common Crawl itself. The conversion from HTML to text is lossy-structure, links, and metadata are stripped-and the extraction quality is modest by modern standards.

As of 2024, there are roughly 100 crawl snapshots, each adding approximately 2-3 billion pages. Naively, this looks enormous: $\sim$240 trillion tokens if you take all snapshots and run minimal filtering (as DCLM-pool does). But the raw quality of random Common Crawl pages is low-boilerplate, spam, near-duplicate content, and non-English text dominate a random sample.

The HTML-to-text conversion step matters more than it looks. Starting from the WARC files and applying a high-quality extractor produces substantially better text than using the WET files directly. The DataComp-LM paper ran a controlled ablation across extraction methods and found that using Trafilatura instead of the raw WET files improved downstream benchmark scores by roughly four points. Your choice of HTML extraction tool is a hyperparameter with measurable impact on model quality.

The appeal of WARC over WET is information density. WET is a blunt conversion: it strips away HTML structure and applies a simple heuristic to identify the “main content” of a page. WARC lets you apply a modern, ML-informed extractor like Trafilatura or jusText, which use statistical models to distinguish article body text from navigation bars, footers, and ads. On a real web corpus, this distinction is not trivial-a significant fraction of WET tokens are boilerplate that would never appear in a well-written document.

The trade-off is cost. Processing WARCs yourself requires storing or streaming tens of terabytes per snapshot, running an extractor at scale, and managing failures in a distributed pipeline. WET is turnkey: download and tokenize. For many teams prototyping a new filtering algorithm, starting with WET is fine. For a production corpus where per-point improvements on benchmarks matter, WET is a ceiling.

There is also a secondary effect: different extractors make different coverage-quality trade-offs. Trafilatura is aggressive at removing boilerplate and retains fewer tokens; jusText is more permissive and retains more, which is important when you are trying to sustain a very large training run. NVIDIA’s Nemotron-CC work explicitly chose jusText over Trafilatura to maximize the token yield for training runs that require well above 6 trillion tokens, accepting slightly noisier extractions in exchange for coverage.

A Worked Example

Consider the DCLM-pool construction: all Common Crawl snapshots up to 2024 are taken, WARC files are processed with Trafilatura, language identification is run to keep primarily English pages, and basic heuristics (minimum document length, removal of mostly non-alphabetic content) are applied. The result is approximately 240 trillion tokens.

Now apply the DCLM-baseline quality filter (a fastText classifier trained to distinguish instruction-like data from generic web text): the pool shrinks to roughly 3.8 trillion tokens-about 1.6% of the original. This aggressive filtering yields high benchmark scores per token, but a 70B model trained for 2 epochs on 3.8 trillion tokens only sees $7 . 6 \times 10^{12}$ training tokens, which is already reaching the data ceiling for that corpus.

Compare to Nemotron-CC starting from the same Common Crawl dumps: by using jusText (more permissive extraction) and a softer ensemble of classifiers rather than a single hard threshold, NVIDIA recovered approximately 6.3 trillion tokens while maintaining quality that outperforms DCLM-baseline on most benchmarks. The arithmetic is simple: same source, different extraction and filtering strategy, 66% more usable tokens.

What You Actually Decide in Practice

• WET vs. WARC: Use WET for prototyping and research where you need fast iteration. Use WARC with a modern extractor (Trafilatura or jusText) for any corpus that will be used to train a model above 7B parameters, where the four-point quality gap is significant.

• Extractor choice: If maximizing benchmark scores per token is the goal, prefer Trafilatura. If you need more tokens to fill a longer training run (e.g., $>$10 trillion tokens), prefer jusText and accept marginally noisier extractions.

• How many snapshots to use: Newer snapshots have more overlap with newer web content. Older snapshots have higher duplicate rates across themselves. Deduplication cost grows super-linearly with the number of snapshots, so most pipelines start with 5-15 recent snapshots and add more only when data quantity becomes the bottleneck.

• robots.txt compliance: Common Crawl does respect robots.txt in principle, but coverage is imperfect. If you are building a corpus for a commercial product, spot-check robots.txt compliance on high-volume domains to avoid legal exposure.

• Incomplete coverage is real: Common Crawl is not the internet. Even applying the same Reddit karma filter used to construct OpenWebText against all of Common Crawl yields only about half the tokens that a direct crawl of those Reddit-linked pages would produce. For domains that matter to your application, augment with targeted crawls.

How This Shows Up in Interviews

• “Walk me through how Common Crawl becomes training tokens.” A strong answer covers: raw crawl produces WARC and WET; you extract text from WARC using a tool like Trafilatura; apply language ID and heuristic filters; deduplicate; optionally apply a quality classifier. Name the 4-point gap between extraction tools.

• “Why not just use the WET files Common Crawl provides?” Key points: WET is a lossy conversion with low-quality boilerplate extraction; modern extractors applied to WARC produce measurably better text; the cost is pipeline complexity and storage, which is worth it above prototype scale.

• Senior follow-up: “DCLM-baseline filters 240 trillion tokens down to 3.8 trillion. Is that a problem?” A senior answer acknowledges the data ceiling this creates for large-model training runs, explains the Nemotron-CC approach of using a softer classifier ensemble to recover 6.3 trillion tokens from the same source, and discusses the coverage-quality trade-off in classifier design. The deeper point is that aggressive quality filtering can become a capacity constraint before compute does.

Key Takeaways

• Common Crawl produces WARC (raw HTML) and WET (pre-extracted text); always prefer WARC with a quality extractor for production corpora.

• Extraction tool choice is a measurable hyperparameter: Trafilatura vs. raw WET is worth roughly four benchmark points in controlled ablations.

• Trafilatura maximizes quality per token; jusText maximizes token yield-choose based on whether your bottleneck is quality or quantity.

• Common Crawl is not a complete snapshot of the web; targeted domain crawls are necessary for coverage of high-value sources.

• Aggressive quality filtering that reduces 240 trillion tokens to 3.8 trillion creates a data ceiling for large training runs; ensemble-based filtering recovers more tokens at comparable quality.

20.1.2 Books, Code, Papers, and Forums

Every serious pre-training mix reserves dedicated weight for curated domain corpora-books, code repositories, academic papers, and Q&A forums-even when web data already totals trillions of tokens. An interviewer asking “why not just filter Common Crawl more aggressively?” is testing whether you understand what these domains provide that filtered web text cannot. This question surfaces regularly in ML infrastructure and research scientist interviews, especially when the topic is domain-specific capability, long-context extension, or instruction-following ability.

Web text is wide but shallow. A typical web page is short, self-contained, and stylistically generic. The corpora described in this subsection are valuable because they provide signal that web pages structurally cannot:

• Books: long-range coherence and narrative structure. A single book can span hundreds of thousands of tokens with thematic continuity, making book corpora the primary tool for long-context extension training. BooksCorpus (7,000 self-published works, now defunct) was used for BERT. Project Gutenberg ($\sim$75,000 public-domain books) is the clean open alternative. Books3, drawn from shadow libraries, was used in the Pile and the original Llama but has since generated substantial copyright litigation.

• Code: formal structure, precise semantics, and executable reasoning chains. The dominant source is GitHub. The Stack (from BigCode) is the open replication: 137 million repositories git-cloned, filtered to permissively licensed files, deduplicated, yielding 3.1 TB of code in over 300 programming languages. Code is commonly cited as improving reasoning and tool-use capabilities beyond programming tasks, though the mechanism is not fully understood.

• Academic papers: dense factual knowledge and structured argumentation. arXiv provides open-access preprints across STEM fields. PubMed Central provides biomedical papers under an NIH open-access mandate. Together they add tens of billions of tokens of high-information-density text that is underrepresented in general web crawls.

• Forums (Stack Exchange, Reddit, Wikipedia): question-answer structure that closely mirrors how users interact with a deployed chat model. Stack Exchange is particularly valuable because votes and comments provide a quality signal that lets you filter aggressively. Wikipedia is used both as a direct training source and as a quality reference (via n-gram language models) to score other documents.

Each domain corpus provides a structural signal-long coherence, formal correctness, QA format, or encyclopedic density-that filtered web text approximates poorly at any reasonable scale.

The core argument is coverage of distributions that web text undersamples. Consider what a random Common Crawl page looks like: short, often list-formatted, frequently e-commerce or news. The long-tail of technical, literary, and mathematical knowledge that makes a model broadly capable lives in books, papers, and specialized forums. A quality classifier trained to select “Wikipedia-like” text from Common Crawl will recover some of this, but it will miss the structural properties (multi-page coherence, formal proof style, multi-turn Q&A) that domain corpora provide natively.

Stack Exchange deserves special attention. A Stack Overflow thread-question, accepted answer, community comments-looks almost identical to a user-assistant chat exchange. Models trained on substantial Stack Exchange data have, in a sense, seen instruction-following examples during pre-training. This is one reason the line between pre-training and post-training data is blurry: the instruction-following distribution is already present in the pre-training mix, just unlabeled.

Code has a different but equally important property: it is almost always semantically valid (it compiles or it doesn’t), it is heavily deduplicated by nature (copy-paste code is easy to detect), and it carries explicit structure (function signatures, docstrings, tests) that exposes relationships between specification and implementation. Training on code improves performance on tasks that require multi-step planning and precise output formatting, even when code itself is not the target.

A Worked Example

The Dolma corpus used to train OLMo (AI2) illustrates how these domains compose in practice. The reported breakdown across $\sim$3 trillion tokens is approximately:

Web text dominates by volume, but code ($\sim$13%) and academic papers ($\sim$3%) punch above their weight: models trained without these domains show notably degraded performance on programming tasks and technical Q&A even though they represent a small fraction of total tokens.

The token counts also reveal a practical constraint: Project Gutenberg at 6 billion tokens is tiny relative to the corpus. It is included not for scale but for structural diversity-long-form text with high coherence that no other source provides at equivalent quality.

What You Actually Decide in Practice

• License filtering for code is mandatory. The Stack and similar pipelines filter to permissive licenses (MIT, Apache, BSD) before any other step. Training on GPL code and releasing the model exposes you to viral license claims; the legal landscape is unsettled enough that teams err on the side of permissive-only. This drops coverage from all of GitHub to roughly one-third of public repositories.

• Wikipedia as a quality anchor vs. training source. Wikipedia is small ($<$0.2% of a typical corpus) but appears in almost every mix. Its dual role-as a direct source of high-quality text and as the reference distribution for quality classifiers-means removing it from training to avoid license concerns would also break most model-based quality filters.

• Books3 and shadow libraries are off the table for commercial models. Meta’s use of LibGen (4M books) is the subject of ongoing litigation. Project Gutenberg (75,000 public-domain books) is the safe alternative; the quality is high but the volume is 50$\times$ smaller.

• Stack Exchange and Reddit licensing changed around 2023. Both platforms moved to require paid licenses for commercial model training. Budget for this or find community-sourced snapshots taken before the policy change. Datasets like Dolma used pre-2023 Reddit dumps; replicating this for a new corpus requires negotiation or a substitute.

• Domain upsampling at mid-training. Even if a domain is well-represented at pre-training scale, it is common to upsample math, code, and long-context data at mid-training to strengthen specific capabilities. The Tulu and OLMo-2 pipelines both do this explicitly, applying 5-20$\times$ upsampling on math and reasoning data relative to their natural frequency in the web corpus.

How This Shows Up in Interviews

• “Why include Stack Exchange in a pre-training corpus? It’s only a few billion tokens.” Key points: volume is not the reason; it is the QA structure, which pre-exposes the model to instruction-following format, and the quality signal from votes that enables aggressive filtering. Contrast with raw Common Crawl where no comparable quality signal exists per document.

• “How do you handle code licensing in a pre-training corpus?” A strong answer covers: filter to permissive licenses (MIT, Apache, BSD) at repository level before extraction; deduplicate aggressively because copy-paste code is common; note that GPL code creates viral license risk that most teams avoid for production models. Mention The Stack as the open benchmark for this approach.

• Senior follow-up: “If you had to cut your corpus from 15 trillion tokens to 3 trillion due to a compute budget reduction, which domains would you preserve and at what ratios?” A senior answer reasons from capability impact rather than volume: preserve code ($\sim$10-15%), academic papers ($\sim$5%), and high-quality filtered web ($\sim$75-80%); cut general Reddit and unfiltered Common Crawl first. Justify by pointing to ablations (Dolma, DCLM) showing that code and papers have outsized impact per token on benchmark performance, while generic web text is the most substitutable component.

Key Takeaways

• Books, code, papers, and forums each provide structural signals-long coherence, formal correctness, QA format, encyclopedic density-that filtered web text cannot replicate at any scale.

• Stack Exchange and Wikipedia appear in nearly every mix not for volume but for structural quality; Wikipedia also doubles as the reference distribution for quality classifiers.

• Code must be filtered to permissive licenses before training; GPL and other copyleft licenses create legal exposure in commercial models.

• Shadow library books (Books3, LibGen) offer substantial volume but are subject to active litigation; Project Gutenberg is the safe 75,000-book alternative.

• Domain data is typically upsampled 5-20$\times$ at mid-training to strengthen targeted capabilities, even when the domain was already present at pre-training scale.

20.2 Evolution of Open Pre-Training Corpora

20.2.1 From C4 and The Pile to RefinedWeb and FineWeb

Every serious interview question about pre-training eventually comes down to one uncomfortable truth: “the internet” is not a dataset. When you first start building corpus pipelines, the naive assumption is that you download Common Crawl, shuffle it, and feed it to the model. What you actually get is mostly boilerplate, spam, and machine-generated SEO text. The history of open pre-training corpora is the history of the community figuring out, dataset by dataset, how to turn Common Crawl into something trainable. Understanding the milestones-C4, The Pile, RefinedWeb, FineWeb-tells you what decisions matter, what mistakes were made, and what the current defaults are. This progression shows up in senior interviews as a framing question: “How did the community’s approach to corpus construction evolve, and why?”

The central challenge is that Common Crawl is simultaneously enormous and noisy. A single monthly snapshot contains roughly 1.4 trillion tokens, but most of it fails to qualify as coherent natural language by any reasonable measure. The filtration problem is: given a very large, very cheap pool of text, how do you maximize the quality-per-token of the subset you actually train on?

The early answer, introduced by C4 (Colossal Clean Crawled Corpus, from the T5 paper in 2019), was purely heuristic: keep lines ending in terminal punctuation, require at least three sentences per document, strip pages containing certain keywords, and discard any text containing curly braces-which removed most code. Starting from one snapshot of approximately 1.4 trillion tokens, these rules yielded roughly 156 billion tokens of English text. The approach is fast, deterministic, and interpretable. Its limitation is equally obvious: wellformed spam still passes, and legitimate domain-specific text (technical documentation, structured prose) often fails.

EleutherAI’s The Pile (2020) took the opposite stance: instead of filtering a single noisy source aggressively, curate 22 deliberately chosen high-quality domains and mix them. The result is an 825 GB dataset (roughly 400 billion tokens) spanning Common Crawl subsets, Books3, arXiv, PubMed Central, GitHub, Stack Exchange, Wikipedia, and others. Each source was selected because domain experts believed it carried signalnot because a classifier said so.

RefinedWeb (2023, from TII) reversed the Pile’s philosophy: it bet that Common Crawl alone, filtered well enough, outperforms any handpicked multi-source blend. Using Trafilatura for HTML extraction, Gopher-style heuristic quality rules, and aggressive fuzzy deduplication, TII produced a 5 trillion token corpus without any model-based classifier. FineWeb from Hugging Face extended this approach to all available Common Crawl dumps, yielding 15 trillion tokens with additional deduplication and light anonymization.

The core progression is from heuristics-on-one-dump, to multi-source curation, to aggressive-heuristics-on-all-dumps-each step trading a different dial between volume, coverage, and quality.

Rule-based filtering has one decisive advantage over model-based filtering: it cannot overfit to a proxy. A classifier trained to find Wikipedia-like text will produce more Wikipedia-like text, not necessarily better pre-training data. In 2020, this was a genuine concern-the models available for filtering were too weak to be trusted for semantic judgments, and aggressive model-based filtering risked stripping out legitimate content from underrepresented dialects and domains that simply did not look like encyclopedia articles.

The Pile’s multi-source design solved coverage differently. Instead of filtering a noisy pool, it sourced directly from known-good repositories: arXiv for scientific reasoning, PubMed for biomedical language, Stack Exchange for technical QA. The trade-off is that assembling and licensing 22 separate sources is expensive and fragile-several Pile sources (Books3, Reddit) have since been removed or restricted due to copyright and API policy changes.

RefinedWeb and FineWeb took the pragmatic position: if you deduplicate thoroughly and apply heuristic rules carefully, you can extract more quality from Common Crawl than any curated multi-source mix, because Common Crawl is simply larger. FineWeb’s 15 trillion tokens dwarfs The Pile by 37$\times$. The deduplication step deserves particular emphasis: fuzzy deduplication using MinHash to find near-duplicate documents consistently improves downstream model quality even when it reduces token count. Models that memorize repeated boilerplate instead of learning novel patterns pay a quality tax that raw token volume cannot compensate for.

A Worked Example

Walk through building a C4-equivalent pipeline from a single Common Crawl snapshot, CC-MAIN-2019-18, which contains approximately 1.4 trillion tokens in WET format.

1. Language filter. Run fastText language identification; keep only documents classified as English with confidence $p > 0 . 65$. This drops roughly 50% of documents. Remaining: $\sim$700 billion tokens.

2. Heuristic rules. Apply C4 rules: drop lines not ending in terminal punctuation, drop documents with fewer than three sentences, drop documents containing { or }, drop documents matching a 400-word blocklist. Remaining: $\sim$165 billion tokens.

3. Exact deduplication. Hash each 3-sentence window with SHA-1; drop any document where more than 30% of windows appeared before. Remaining: $\sim$156 billion tokens.

That is the C4 slice: one snapshot, three passes, 11% retention. Now extend to FineWeb: repeat across all 100 Common Crawl snapshots (roughly 140 trillion token raw pool), replace step 2 with Gopher rules, add MinHash fuzzy dedup with a Jaccard threshold of 0.8, and you converge toward 15 trillion tokens of retained text-the FineWeb baseline.

What You Actually Decide in Practice

• WET vs. WARC extraction. Using pre-extracted WET files is faster but delivers 3-4 benchmark points lower than re-extracting HTML with Trafilatura or jusText. Pay the compute cost of WARC extraction once; you will not redo the corpus every month, and the quality gain is permanent.

• Deduplication scope. Within-snapshot exact dedup is cheap. Cross-snapshot dedup catches repeated boilerplate (news syndication, Wikipedia mirrors) but requires storing hundreds of billions of hashes. For most teams, within-snapshot exact dedup plus cross-snapshot MinHash fuzzy dedup is the right balance.

• Multi-source vs. single-source. If your targets are domain-specific (biomedical, legal, scientific), include those sources explicitly-web data undersamples technical literature by roughly $10 \times$ relative to professional usage. For general language, FineWeb or a filtered Common Crawl is sufficient.

• Rule-based vs. model-based. Pure heuristics are reproducible and fast; use FineWeb as your base and layer model-based filters on top for targeted quality improvements. Mixing both is now standard in production.

• Token budget alignment. At 7B parameters, roughly 1 trillion high-quality tokens hits the Chinchillaoptimal frontier. At 70B, that rises to 10 trillion. Choosing a dataset with fewer tokens than your compute budget forces multi-epoch training, which hurts quality; an oversized dataset means you stop before the model is saturated.

How This Shows Up in Interviews

• “Walk me through how you would build a pre-training corpus from Common Crawl.” A strong answer names the four stages in order: extract (WET vs. WARC, choice of tool), language filter, quality filter (heuristic rules and/or classifier), deduplication (exact then fuzzy). Mention approximate retention rates at each step-11% retention for C4-style filtering on a single snapshot is a concrete number worth having.

• “Why does The Pile use 22 sources instead of just filtering Common Crawl harder?” Coverage: Common Crawl systematically underrepresents technical domains. PubMed, arXiv, and Stack Exchange are not well-crawled, so model-filtered web data misses them regardless of filter quality. A strong answer also notes that this advantage erodes as legal and API access to those sources becomes restricted.

• “What is the difference between C4 and FineWeb? When would you use one over the other?” C4 is a single-snapshot, heuristic-only, 156-billion-token dataset-useful for ablations and reproducibility. FineWeb is all-dumps, 15 trillion tokens, heuristic rules plus fuzzy dedup-use it for any real training run where you want a strong general-purpose baseline with more than enough tokens for a 70B model.

• Senior follow-up: “FineWeb outperforms C4 on benchmarks. Is that because of more tokens or better deduplication?” The senior answer isolates the variables: FineWeb uses $\sim$96$\times$ more tokens but also applies MinHash deduplication, which C4 lacks. Controlled ablations in the FineWeb release show that deduplication alone accounts for most of the gap on knowledge-intensive benchmarks, while the additional token volume helps primarily on few-shot generalization tasks. The practical lesson is that deduplication is higher leverage than raw volume once a corpus exceeds a few hundred billion tokens.

Key Takeaways

• C4 established the heuristic-filtering template: punctuation rules, sentence-count thresholds, blocklists applied to one Common Crawl snapshot, yielding 156 billion tokens at 11% retention.

• The Pile showed that multi-source curation outperforms filtered web alone when domain coverage matters, but the approach is fragile to copyright and API restrictions that have since removed several of its key sources.

• RefinedWeb and FineWeb validated the counter-bet: aggressive fuzzy deduplication plus heuristics applied to all Common Crawl dumps produces a superior general-purpose corpus without model-based filtering, at 5-15 trillion tokens.

• Fuzzy deduplication (MinHash, Jaccard $\sim$0.8) is higher-leverage than additional raw tokens once a corpus exceeds a few hundred billion tokens-it is now standard practice in every serious pipeline.

• HTML extraction tool choice (Trafilatura or jusText vs. pre-extracted WET) shifts benchmark scores by 3-4 points and is one of the cheapest, highest-return decisions in the entire pipeline.

20.2.2 DCLM, Nemotron-CC, and the Modern Recipe

The 2024 DataComp-LM (DCLM) paper changed the default assumptions for every team building a serious pre-training pipeline. Before it, the community had largely avoided machine-learning classifiers for filtering pretraining data, on the grounds that such filters introduce hard-to-audit biases and may strip out underrepresented content. DCLM settled the question empirically through a controlled benchmark competition: a fastText classifier trained on instruction-style positive examples, applied aggressively to Common Crawl, outperforms every prior heuristic baseline on standard evaluations. Nemotron-CC from NVIDIA followed with a complementary finding-aggressive quality filtering preserves quality but destroys token volume, and at the scale required for a 70B-class model you need both. These two datasets now define the upper boundary of what is publicly achievable in open pre-training data, and understanding their pipelines in detail is expected knowledge for any senior ML engineer working on pre-training.

DCLM frames corpus construction as a machine-learning competition: given a fixed compute budget for a 7B-parameter model, which data pipeline produces the best benchmark scores? To make this repeatable, the team first processed all available Common Crawl dumps through a unified extraction and MinHash deduplication pipeline, producing DCLM-pool: 240 trillion tokens of deduplicated English web text. The competition then asks: what quality filter, applied to DCLM-pool, yields the best downstream model?

Their winning recipe is a fastText binary classifier. The positive training set combines two sources:

• OpenHermes-2.5: a collection of GPT-4-generated instruction-following examples covering diverse tasksquestion answering, reasoning, code, summarization.

• ELI5: the Reddit “Explain Like I’m 5” subreddit, where community members write detailed, accessible explanations in response to curious questions. Negative examples are sampled uniformly from RefinedWeb-not garbage, but less curated than the positives. The classifier score is computed for every document in DCLM-pool; keeping documents above a threshold of 0.5 produces DCLM-baseline:

3.8 trillion tokens retained $\approx \qquad \approx 1 . 6 \%$ of the pool

DCLM-baseline outperforms RefinedWeb by approximately 3 percentage points on the core benchmark suite (HellaSwag, ARC, MMLU) when training a 7B model to 1 trillion tokens.

Nemotron-CC from NVIDIA accepted DCLM’s quality signal but rejected its extreme selectivity. Their argument: training a 400B-parameter model to Chinchilla optimality requires on the order of 8-10 trillion tokens. DCLM-baseline at 3.8 trillion is insufficient. Their response was to ensemble multiple quality signals-the DCLM fastText score, an educational-value classifier distilled from a large Nemotron model, and document length-and then stratified-sample across score buckets rather than hard-thresholding at the top. The result was Nemotron-CC: 6.3 trillion tokens, outperforming DCLM-baseline on benchmarks while providing 66% more training tokens.

A further Nemotron-CC contribution was LLM-assisted document transformation. Low-quality retained documents were rewritten into cleaner prose by an LLM. High-quality documents had synthetic instruction pairs generated from them-for example, a Wikipedia article was converted into a “Summarize this” task, a “List the key facts” task, and a factual question-answer pair. This deliberately blurs the boundary between pre-training and post-training data, with the explicit goal of giving the model early exposure to instruction-following structure.

The modern recipe is: DCLM-pool for extraction and dedup, a fastText or LLM-distilled classifier for quality scoring, stratified sampling across score strata for volume control, and LLM rephrasing for the long tail of retained documents.

Using instruction-style data as positive examples for a pre-training quality classifier is counterintuitive-you are not training on instructions, you are looking for web documents that resemble instructions. The insight is that well-formed question-and-answer content is a strong proxy for coherent, informative language. ELI5 posts and OpenHermes examples are explanatory, specific, and well-structured; a document that looks like them tends to be substantive rather than spammy.

The competing approach-using Wikipedia as the quality anchor, as CCNet and GPT-3 did-produces data that looks like an encyclopedia. That is excellent for factual recall but underweights reasoning chains, instructional prose, and conversational question-answering. DCLM’s ablations show that the instruction-anchored positives outperform the Wikipedia-anchored approach by 1-2 points even when controlling for token count.

Nemotron-CC’s stratified sampling strategy addresses a genuine production problem. Hard-thresholding at a high quality score is like accepting only the top decile of candidates: you get quality, but you sacrifice diversity in the lower nine deciles, some of which contain irreplaceable domain-specific content. Bucketing by score and sampling proportionally preserves quality distribution while maintaining volume. The 1 trillion token high-quality Nemotron-CC subset (the top two score buckets) outperforms the full 6.3T on per-token quality metrics, confirming that quality and volume trade off rather than compound; the extra tokens are worth including only when your training budget demands them.

A Worked Example

Walk through the DCLM pipeline for a 7B model targeting 1 trillion tokens of training.

Step 1: Obtain DCLM-pool. DCLM-pool is publicly available on Hugging Face. It contains 240 trillion deduplicated English tokens, already MinHash-deduplicated across all Common Crawl dumps. Do not rebuild this yourself; the extraction and dedup engineering is the most expensive part and is already done.

Step 2: Train the fastText classifier.

Positive set : 300K docs from OpenHermes-2.5
               + 50K docs from ELI5 subreddit
Negative set : 350K docs sampled from RefinedWeb
Labels       : __label__positive / __label__negative
Training     : fastText supervised, 5 epochs, lr=0.1
Model size   : ~2 GB
Train time   : <10 minutes on a single CPU machine

Step 3: Score DCLM-pool. Run the classifier over 240 trillion tokens. At $\sim$1 million tokens per second per CPU core, scoring the full pool requires $\sim$240 million CPU-core-seconds, or roughly 2,800 CPU-corehours-about 28 hours on a 100-core cluster.

Step 4: Filter. Keep documents with fastText positive-class $\text{score} > 0 . 5$. Empirically this retains 1.4- 1.6% of documents, yielding 3.4-3.8 trillion tokens. Sample 1 trillion uniformly from this retained set for your training run.

Step 5: Validate with a proxy model. Train a 1B-parameter proxy on 10 billion token samples drawn from (a) the retained set and (b) FineWeb. A 1-2 point improvement on HellaSwag confirms the classifier is working. If you see no improvement, check that your positive-set format matches typical web document encoding (UTF-8, not HTML entities).

Total additional cost over downloading DCLM-pool: approximately 3,000 CPU-core-hours for scoring. This is negligible compared to GPU-hours for the actual 7B training run.

What You Actually Decide in Practice

• Start from DCLM-pool, not raw Common Crawl. Reproducing the extraction and deduplication pipeline from scratch takes weeks of engineering. DCLM-pool is public, covers all dumps through mid- 2024, and the only cost is download bandwidth. Reserve engineering time for the classifier.

• Hard-threshold for models under 70B; stratified-sample for larger. If your token budget is under 5 trillion, a hard threshold at the DCLM fastText score of 0.5 is sufficient. Above 5 trillion-which is where 70B$+$ models live at Chinchilla optimality-use stratified sampling or lower the threshold, consciously trading some quality for volume.

• Do not use GPT-4-generated data as positives in a production classifier. It violates OpenAI terms of service. Use open-weight model outputs (Llama 3, Qwen, Mistral) or human-curated instruction datasets under permissive licenses (e.g., FLAN, Dolly, OpenHermes under its Apache-2.0 variant).

• LLM rephrasing is worth the cost for the bottom quartile only. Rephrasing the lowest-scoring 25% of retained documents with a 7B model costs roughly $0 . 25 \times 3 . 8$ T $\approx 950$ billion tokens of generation. Budget this only if those documents cover rare domains not well-represented in the top tier; otherwise the hallucination risk outweighs the coverage gain.

• Use Nemotron-CC-1T as your default single-source option for a 7B run. It is the highest-quality public corpus as of mid-2024 at a token count that fits comfortably within a 7B Chinchilla-optimal budget, and its benchmark numbers are well-documented.

How This Shows Up in Interviews

• “How would you use a quality classifier to filter a 200-trillion-token web corpus?” Name the pipeline: define positive examples (instruction-style data), define negatives (lightly filtered web), train fastText (fast, interpretable, $<$2 GB model), score the pool in parallel CPU jobs, threshold at a target retention rate. Mention that 1.4-1.6% retention at a threshold of 0.5 yields $\sim$3.8 trillion tokens-enough for a 7B model.

• “What makes DCLM-baseline better than FineWeb if both come from Common Crawl?” The classifier. FineWeb uses only heuristic rules; DCLM adds a fastText classifier trained on instruction-like positives. The DCLM paper shows a 3-point improvement on core benchmarks at 7B scale. Both apply MinHash deduplication, but the classifier is the dominant factor in the quality gap.

• “What is the trade-off between DCLM-baseline and Nemotron-CC?” DCLM-baseline: higher average quality per token, 3.8 trillion total tokens-constrains training runs for large models. Nemotron-CC: slightly lower average quality per token across the full 6.3T, but includes a 1T high-quality subset that matches or exceeds DCLM quality while providing 66% more tokens for training if needed. Choose DCLM for 7B; choose Nemotron-CC full for 70B$+$ where token volume limits the training run.

• Senior follow-up: “Nemotron-CC uses LLM rephrasing to recover low-quality documents. What could go wrong, and how would you detect it?” The core risk is hallucination: the rewriting model invents facts or systematically injects its own biases into the training corpus. Three detection strategies: (a) perplexity divergence-rephrased documents with perplexity far below their source indicate fabrication; (b) n-gram overlap checks-if rephrased text deviates dramatically from the source, flag for review; (c) factual benchmark regression-if adding rephrased documents degrades TriviaQA or NaturalQuestions relative to the clean baseline, the rephrasing model is corrupting factual knowledge. In production, cap the rephrased fraction at 10-15% of total tokens and treat factual benchmarks as a canary before scaling up.

Key Takeaways

• DCLM-pool covers all Common Crawl dumps through mid-2024 (240 trillion deduplicated tokens); DCLM-baseline applies a fastText quality classifier to reduce it to 3.8 trillion tokens-roughly 1.6% retention-outperforming prior heuristic baselines by 3 points at 7B scale.

• Instruction-like data (OpenHermes, ELI5) as positive examples for the quality classifier outperforms Wikipedia-anchored proxies; the intuition is that explanatory, question-answering prose is a better proxy for useful pre-training content than encyclopedia-style writing.

• Nemotron-CC addresses DCLM-baseline’s volume limitation through ensemble classifier scoring, stratified sampling across score buckets, and LLM-assisted rephrasing, yielding 6.3 trillion tokens with quality competitive with DCLM-baseline.

• For a 7B model, start with DCLM-baseline or Nemotron-CC-1T; for a 70B$+$ model, use Nemotron-CC full (6.3T) or lower the DCLM threshold to recover token volume at the cost of some quality dilution.

• The key engineering insight of the modern recipe: spend CPU cycles on classification (cheap, parallelizable), spend GPU cycles only on model training-the classifier itself is a fastText model that trains in under ten minutes.

20.3 Legal and Ethical Considerations

20.3.1 Copyright, Licenses, and Fair Use

Copyright is the central legal friction in assembling a pre-training corpus, and it surfaces in almost every senior-level data interview. The question “can you legally train on Common Crawl?” sounds simple, but answering it well requires understanding three distinct concepts that most engineers conflate: what copyright covers, how licenses grant permission, and when fair use provides a defense without a license. Getting this wrong does not just expose the company to litigation-it shapes which data sources you can include in an open-weight release versus a closed commercial product.

Under the US Copyright Act of 1976, copyright attaches automatically to any original work of authorship fixed in a tangible medium of expression. Registration is not required; the moment someone publishes a blog post, that post is copyrighted. This means the overwhelming majority of web content that ends up in Common Crawl is copyrighted, regardless of whether the page displays a copyright notice.

Copyright protects expression, not ideas. You cannot copyright the algorithm for computing attention; you can copyright the specific code implementing it. Facts-names, dates, addresses-are not copyrightable, which is why a phone book’s content is in the public domain even though a sufficiently creative arrangement of that content might not be. This distinction matters for training data: corpora heavy in factual encyclopedic text (Wikipedia, arXiv abstracts) sit in a more defensible position than corpora heavy in fiction, journalism, or creative prose.

There are two legal paths to use copyrighted material. The first is a license: a contract in which the rights holder grants you permission under specified terms. Creative Commons licenses (CC-BY, CC-SA, CC-NC, etc.) are a standardized form of pre-granted license-Wikipedia is CC-BY-SA, so anyone may use its text with attribution and share-alike obligations. Commercial licenses negotiated directly with platforms (Reddit-Google, Stack Exchange-OpenAI) work the same way but are bespoke and paid. The second path is fair use: a statutory defense that allows use of copyrighted material without a license when four factors favor it: (1) the purpose and character of the use (commercial vs. educational, transformative vs. reproductive); (2) the nature of the copyrighted work (factual vs. creative); (3) the amount used relative to the whole; and (4) the effect on the market for the original.

Whether training an LLM on web-scraped text constitutes fair use is unresolved law, but the strongest argument turns on the “transformative use” prong: the model extracts statistical patterns across billions of tokens, not a copy of any particular work.

The transformative-use argument draws force from Authors Guild v. Google (2d Cir. 2015), which held that Google Books’ systematic digitization and snippet display of millions of copyrighted books was fair use because it served a fundamentally different function than reading the books-it enabled search and discovery. LLM training proponents argue the analogy holds: the model does not reproduce the training corpus; it compresses statistical regularities into weights that support generation of new text.

The tension is that LLMs can memorize. Carlini et al. showed that training data can be extracted verbatim from GPT-2 given the right prompt, and subsequent work demonstrated the same for larger models. A model that can regurgitate entire paragraphs of a copyrighted novel weakens the transformative-use defense considerably, because the infringement is no longer abstract-the expression itself is reproduced. This is why deduplication and memorization audits are now standard pipeline steps, not just quality hygiene.

The commercial/non-commercial distinction also matters asymmetrically. Academic researchers running non-commercial experiments are more insulated than a company selling API access. Open-weight releases sit in a legally awkward middle: the weights are “open,” but the training data disclosure that accompanies them makes the copying more legible to plaintiffs. Closed commercial models paradoxically enjoy better legal cover on this dimension because the training data is never disclosed.

A Worked Example

Consider the trajectory of Books3, a dataset of roughly 200,000 copyrighted books scraped from Library Genesis, a shadow library. Books3 appeared in the Pile (2020) and was later incorporated into Llama 1’s training mix. In 2023, a group of authors including Richard Kadrey, Sarah Silverman, and Christopher Golden sued Meta, alleging that Llama 1’s use of Books3 constituted copyright infringement.

Meta’s primary defense was transformative use: training compresses millions of books into 7 billion parameters that do not store or reproduce any individual work. The plaintiffs countered that the model can reproduce passages verbatim and that it effectively substitutes for buying the books, harming the market. The district court dismissed several claims but allowed some to proceed. In parallel, Books3 was removed from the Pile’s hosted version and Llama 2 quietly dropped it from its data card.

The practical lesson: the risk profile of a dataset is not just “is this copyrighted” but “is it from a source whose rights holders are organized, motivated, and able to sue.” Published authors are a concentrated, articulate group. Individual bloggers are not. This asymmetry-not legal principle-explains why Books3 triggered litigation while the vast majority of Common Crawl’s copyrighted content has not.

As a contrast, Project Gutenberg’s 75,000 books are predominantly in the public domain (copyright has expired, typically 95+ years after publication in the US). Using them is legally safe, and they provide long-document text that books corpora were specifically valued for in the BERT era.

What You Actually Decide in Practice

• Public domain$\neq$ freely usable scans. A 19th-century novel is in the public domain, but a publisher’s typeset PDF of that novel may carry a new copyright on the layout and OCR corrections. Use raw Project Gutenberg text, not publisher PDFs.

• Creative Commons licenses have variants. CC-BY and CC-SA allow commercial use; CC-NC (non-commercial) does not. If your pipeline ingests CC content, track the license variant per document. Mixing NC-licensed content into a commercial training run is a genuine legal risk.

• Open-weight releases attract more scrutiny than closed products. If you publish your data card, you are handing plaintiffs a roadmap. For open-weight models, prefer datasets with clear provenance: FineWeb (Common Crawl, ToS-respecting crawl), DCLM-baseline, The Stack (permissively licensed code only).

• Memorization audits belong in the pipeline. Run a membership-inference probe or verbatim-extraction test against held-out copyrighted text before a model release. A model that can reproduce more than a few sentences of a copyrighted work verbatim is a liability.

• Shadow libraries are a hard no for commercial products. LibGen has 4 million books; it also has a litigation history. The token-per-lawsuit ratio looks attractive until the first discovery subpoena arrives.

How This Shows Up in Interviews

• “Walk me through whether training on Common Crawl is legal.” Strong answer: most Common Crawl content is copyrighted; the legal path is fair use, specifically the transformative-use prong from Authors Guild v. Google; the argument weakens if the model memorizes verbatim text; commercial vs. non-commercial use affects the analysis; no court has definitively ruled on LLM training specifically.

• “What’s the difference between a public dataset and a legally safe dataset?” Strong answer: “public” means accessible without authentication; “legally safe” means you have a license or a defensible fair-use argument. YouTube videos are public but scraping them violates ToS. Wikipedia is publicly licensed (CC-BY-SA). Common Crawl is public but largely copyrighted content used under implicit fair-use reliance.

• “How would you audit a proposed training dataset for legal risk before a model release?” Strong answer: categorize each source by license type; flag shadow-library or paywalled content; run memorization probes on high-risk sources; check whether rights holders in each category have litigation history; produce a data card that documents provenance and any exclusions.

Senior follow-up: “What changes in your analysis if this is an open-weight release versus a closed API?” Senior answer: open-weight releases require full data disclosure, which lowers plausible deniability and makes it easier for plaintiffs to identify infringing sources; commercial use eliminates the educational fairuse factor; the calculus therefore shifts toward more conservative sourcing (CC-licensed, public domain, licensed data) for open releases, while closed commercial models rely more heavily on fair-use arguments they never have to articulate publicly.

Key Takeaways

• Nearly all web content is automatically copyrighted; the question is not whether it is copyrighted but whether you have a license or a fair-use defense.

• Fair use turns on four factors; for LLM training the strongest factor is transformative use, and the weakest point is verbatim memorization.

• Creative Commons licenses differ by variant: CC-BY and CC-SA allow commercial use; CC-NC does not.

• Shadow libraries (LibGen, Books3) carry concrete litigation risk for commercial products; public-domain sources (Project Gutenberg) and permissively licensed code (The Stack) are the safe alternative.

• Open-weight model releases face higher legal exposure than closed APIs because data provenance is disclosed, making memorization-based claims easier to assert.

20.3.2 Terms of Service and Data Governance

Copyright is not the only legal friction in data pipelines. Even when fair use provides a plausible defense for the content of a web page, a separate body of law-contract and computer-fraud statutes-governs how you obtained that content. Engineers who understand copyright but treat platform terms of service as advisory text are operating with a dangerously incomplete picture. At staff and principal level, interviewers test whether you can reason about the full legal surface of a data decision, not just the copyright slice of it.

When you visit a website, you implicitly accept its terms of service (ToS). Most platform ToS prohibit automated scraping, redistribution of content, and use of data to train competing AI systems. Violating ToS is not a copyright infringement-it is a breach of contract, and depending on how the data was accessed, potentially a violation of the Computer Fraud and Abuse Act (CFAA). These are distinct legal theories with distinct remedies and different parties who can sue.

robots.txt is a machine-readable file at the root of a web server that signals which crawlers are permitted to access which paths. It is not legally binding-there is no statute that makes ignoring robots.txt illegal by itself-but courts have treated a site’s robots.txt as evidence of the operator’s intent, which strengthens a CFAA or breach-of-contract argument when a crawler ignores it. The major frontier-lab crawlers (GPTBot, ClaudeBot, GoogleBot for AI training) are all explicitly named in robots.txt files across major news, publishing, and social media sites, with access typically denied.

A data licensing agreement is the commercial resolution to this friction: the platform grants a contractual right to use its data for AI training in exchange for payment or a cross-licensing deal. Examples include the Reddit-Google agreement (2024), the OpenAI-Stack Exchange agreement, and the OpenAI-Shutterstock agreement. These deals emerged as platforms recognized that their data had become a scarce input to a high-value industry and that they could monetize it rather than watch it be scraped for free.

The practical constraint this imposes is stark: there is no way to get a ToS license for the entire internet, which means any corpus that relies on broad web crawl is necessarily operating under a fair-use assumption while simultaneously accepting the contractual risk of ToS violation.

The governance structure that has emerged-frozen Common Crawl snapshots plus separately licensed high-value verticals-reflects the only feasible way to assemble a large, legally defensible corpus at scale. Common Crawl performs the crawl on your behalf and explicitly intends the data for research use, which provides one layer of insulation. Separately licensing Reddit for social dialogue data, Stack Exchange for QA data, and a news aggregator for current-events coverage fills the gaps where the high-value signal is concentrated and the rights holders are organized enough to enforce their ToS.

Data poisoning introduces a governance dimension that is easy to underestimate. Carlini et al. demonstrated a timing attack against Wikipedia: an adversary can inject a malicious edit to a Wikipedia article in the window between the edit and the next editorial rollback, timed to coincide with Common Crawl’s periodic dump. The injected text enters the dump, gets included in downstream training data, and is rolled back on Wikipedia before anyone notices. The result is a model trained on attacker-controlled text, which can be used to steer model behavior-for example, associating a trigger phrase with a specific sentiment or behavioral policy. The fix is partial: Wikipedia has hardened its edit policies, and some pipeline operators now diff their Wikipedia snapshots against a trusted reference. But the broader point stands: data provenance is a security surface, not just a legal one.

A Worked Example

Consider a concrete pipeline decision around YouTube. Millions of YouTube videos carry Creative Com-mons licenses (CC-BY), making the content legally usable without a separate agreement. An engineer might reasonably conclude: “I have a license to the content, so I can download and transcribe these videos for training.” This is wrong.

YouTube’s ToS explicitly prohibits automated downloading of video or audio, regardless of the content’s license. The Creative Commons license governs what you can do with the content once you have it lawfully; the ToS governs how you obtain it. Downloading programmatically without YouTube’s permission is a breach of contract, and YouTube has sent DMCA-adjacent takedown notices and pursued injunctions against services built on this basis (e.g., hiQ Labs v. LinkedIn, though that case turned on CFAA specifically).

The correct path for YouTube audio is either: (1) negotiate a data licensing agreement with Google, as some frontier labs are rumored to have done; (2) use the official YouTube Data API within its rate limits (which are set too low for training-scale data collection); or (3) use transcripts that were lawfully obtained and redistributed under a license (several academic ASR datasets exist in this category). The “it’s Creative Commons” shortcut collapses the distinction between content rights and access rights.

What You Actually Decide in Practice

• Treat robots.txt as legally meaningful, not just advisory. Respecting Disallow directives does not give you immunity, but ignoring them is affirmative evidence against you in both CFAA and breach-of-contract claims. Crawl engines should parse and honor robots.txt by default; build in a crawl-delay header to avoid overload arguments as well.

• Distinguish access rights from content rights. Before adding a source, ask two separate questions: (a) Is the content licensed for this use? (b) Did we obtain it through a channel permitted by the platform? Both must be yes. Creative Commons answers (a); the ToS answers (b).

• Use frozen snapshots to avoid ToS time-bombs. Reddit and Stack Exchange changed their APIs in 2023 mid-production for many teams. If you collected data before the ToS change, the legal exposure is frozen at the time of collection. Snapshot-based pipelines (Common Crawl dumps, Wikipedia XML dumps with a dated version) are more defensible than live API scraping that crosses ToS update boundaries.

• Track provenance at the document level. For every document in your corpus, record the Common Crawl snapshot ID, the retrieval date, and the source URL. This is not just good hygiene-it is the audit trail you need if a rights holder demands removal (a “right to erasure” request under GDPR or a DMCA notice), and it lets you exclude specific domains post-hoc without reprocessing everything.

• Implement a data-poisoning audit for high-trust sources. For sources included specifically because of their quality (Wikipedia, arXiv), maintain a checksum or diff against a reference snapshot. An edit that appeared only in one Common Crawl dump but not in the live site warrants manual review before that document enters the training corpus.

How This Shows Up in Interviews

• “How would you design a legally defensible data collection pipeline for a commercial LLM?” Strong answer: layer the strategy-use Common Crawl as the broad web base (crawl performed by a third party, data intended for research/AI use); negotiate licenses for the highest-value verticals (code, books, QA); exclude sources whose ToS explicitly prohibit AI training use; respect robots.txt in any first-party crawling; maintain per-document provenance logs for erasure requests.

• “What is data poisoning and how would you defend against it in your pipeline?” Strong answer: data poisoning is the injection of adversarial text into training data to steer model behavior; for Wikipedia, the attack surface is the dump-timing window between edit and rollback; mitigation is diffing snapshot content against a trusted live reference, flagging documents that appeared transiently; for Common Crawl, domain-level reputation signals can exclude known spam or compromised domains.

• “A rights holder sends a takedown notice claiming their content is in your training set. Walk me through how you respond.” Strong answer: use provenance logs to identify which documents originated from the claimant’s domain; assess whether a license or fair-use argument applies to that source; if neither, remove the documents from the dataset, retrain affected checkpoints if the content contributed meaningfully, and document the removal.

Senior follow-up: “Can you actually “unlearn” a document from a trained model, and how would you handle this operationally at scale?” Senior answer: machine unlearning is an active research area but not yet production-reliable-gradient-based unlearning methods exist but cannot provide the same guarantees as simply not training on the data in the first place. In practice, the operational answer is (1) maintain mod-ular training checkpoints keyed to data provenance so a corpus can be reconstructed without the disputed content and a fresh training run initiated from the last clean checkpoint; (2) for models already deployed, implement output filters that suppress verbatim reproduction of the claimed content while the longer-term retraining proceeds; (3) accept that this is expensive, which is why upstream provenance hygiene is worth the investment.

Key Takeaways

• ToS violations are a breach-of-contract risk distinct from copyright; both must be evaluated independently for any data source.

• robots.txt is not legally binding but is treated as evidence of intent; always honor it in first-party crawls.

• Creative Commons licenses answer “can I use this content?” not “can I collect it from this platform?”-the ToS answers the second question.

• Document-level provenance logs (snapshot ID, retrieval date, source URL) are the operational prerequisite for DMCA compliance, GDPR erasure requests, and data-poisoning audits.

• Data poisoning is a real attack surface: snapshot-diffing against a trusted reference is the practical mitigation for high-trust sources like Wikipedia.