Chapter 5: Data, Privacy, and the Legal Surface

00-START-HERE · 01-READING-ROUTES · 04_what_the_llm_brings_and_what_it_breaks · 06_scaling_laws_and_the_economics_of_retrieval_vs_parameters This chapter prepares you to explain how Retrieval-Augmented Generation (RAG) architecture changes the cost, evidence, and reversibility of data decisions without claiming that it resolves the underlying legal questions.

TL;DR

• A training pipeline destroys provenance in stages, so a trained model cannot answer document-level questions about its own corpus.
• Alignment can suppress memorized text in ordinary prompts, but it does not delete that text from the weights.
• Duplication makes a sequence easier to extract, and a divergence prompt can move generation outside the behavior covered by alignment.
• Copyright exposure starts with acquisition and copying, while output filters only test a limited class of later emissions.
• Exact unlearning targets equivalence to training without the target data; this chapter uses retraining as its baseline. Approximate unlearning is cheaper, but it does not solve target enumeration or guarantee zero side effects.
• Repeated approximate deletion can compound small capability losses into a large system loss.
• SILO keeps low-risk content in weights and revocable content in a keyed datastore, where removal is selectable and verifiable. The datastore boundary changes the available remedy. It does not by itself make copying lawful or eliminate legal risk.

The story

Imagine a library that must answer a publisher’s request to remove a collection. The receiving clerk starts with shipping records, dates, and license labels. The scanning clerk turns each book into plain text and throws away much of that paperwork. The cleaning clerk removes duplicates and low-quality pages, but keeps whole documents that can still be sampled. The shredding clerk then cuts every document into tokens, mixes the pieces, and packs them into anonymous boxes. Those boxes feed an apprentice who learns patterns and stores them in memory. The library can no longer point to the publisher’s books inside the apprentice’s memory. It can only ask questions and observe what the apprentice recites. A manners coach later teaches the apprentice to refuse sensitive requests. The coaching changes normal behavior, but it does not remove remembered names, addresses, or passages. An attacker can force the apprentice outside the coached routine with a long repetition task. Once there, the apprentice may recite heavily repeated material because repetition made that material easy to learn. The library can compare a recited passage with a known book, but that only detects a literal match. It does not reconstruct how the copy was acquired. It also does not settle broader questions about non-literal similarity. The director next proposes memory surgery. Exact surgery means training a new apprentice without the collection. Approximate surgery punishes the apprentice for repeating examples that a probe found. The probe cannot reveal every relevant example, and the surgery can also damage unrelated knowledge stored in shared memory. Doing this once may look cheap. Doing it for hundreds of requests can steadily erode the apprentice’s capability. The better library design uses two shelves. Permanent, low-risk material may train the apprentice. Revocable material stays in a cataloged reading room. At answer time, a librarian retrieves relevant pages from that room and gives them to the apprentice. When a publisher revokes a license, the librarian selects the publisher’s rows, blocks them immediately, rebuilds the index later, and verifies that the pages no longer appear. The library still needed a lawful basis to make and hold the reading-room copy. The architectural gain is narrower and concrete: the library preserved a key, a delete operation, and evidence that the removal took effect.

Decoder table

Core mechanics

Mathematical conventions and assumptions

Storage uses decimal units unless explicitly marked binary: $1\,\mathrm{kB}=10^3\,\mathrm{B}$, $1\,\mathrm{GB}=10^9\,\mathrm{B}$, $1\,\mathrm{TB}=10^{12}\,\mathrm{B}$, and $1\,\mathrm{GiB}=2^{30}\,\mathrm{B}$. Keep full precision during calculation and round only the reported result; $\approx$ denotes a rounded value or modeling approximation. For the worked compute estimates, let $N$ be parameter count, $T$ the tokens processed, $P_{\mathrm{GPU}}=3.4\times10^{14}\,\mathrm{FLOP\,s^{-1}}$ the assumed sustained throughput per GPU, $r=\$2.50$ per GPU-hour, and $G$ the number of equally utilized GPUs. Then

These are idealized FLOP-based estimates, not measured latency or invoices. They omit attention-length effects, memory-bandwidth limits, communication, queuing, retrieval I/O, and other overhead; using one throughput for prefill, decode, and training is an explicit simplification. Perfect multi-GPU scaling is assumed where wall time is quoted. Corpus statistics and empirical attack rates are reported inputs, not conclusions proved by the arithmetic. A contractual $30$-day window is also distinct from one calendar month.

5.1 What is actually in the pre-training corpus

The lossy funnel

A monthly Common Crawl snapshot begins at roughly $20\,\mathrm{TB}$ of scraped text. Raffel et al. report that filtering leaves about $750\,\mathrm{GB}$. The filters include English-only classification, deduplication, terminal-punctuation rules, code-brace rules, and blocklisted terms. The surviving byte fraction is $750/20{,}000=0.0375=3.75\%$. The cleaned corpus contains about $1.56\times10^{11}$ tokens, implying approximately $4.81\,\mathrm{B/token}$. The conversion matters because corpus reports alternate between bytes and tokens.

A run of $15.6\times10^{12}$ tokens corresponds to about $75\,\mathrm{TB}$ under that byte conversion. Bidirectional Encoder Representations from Transformers (BERT) used a $16\,\mathrm{GB}$ training corpus according to Liu et al. The implied ratio is $75{,}000/16=4{,}687.5\approx4{,}700$, which the source places across six years. Review, licensing, and consent processes did not scale at the same rate. Without the funnel view, an engineer may call the corpus “the internet” and miss every stage that changed it.

Composition and extractor choice

The Pile contains $825\,\mathrm{GiB}$ from $22$ sources. Its sources include web text, PubMed Central, GitHub, arXiv, FreeLaw, Stack Exchange, Wikipedia, and Books3. Using the reported $101\,\mathrm{GiB}$ for Books3 gives $101/825\approx12.2\%$ of The Pile by bytes. Dodge et al. found that patents.google.com was the largest single source domain in C4. That result undercuts the casual assumption that Wikipedia or news dominates the corpus. Gehman et al. found $2\%$ to $4\%$ of documents above a toxicity threshold after cleaning. If the affected documents have the same average length as the corpus, a $2\%$ document share would also be a $2\%$ byte share, or $0.02\times750\,\mathrm{GB}=15\,\mathrm{GB}$, roughly BERT’s full training corpus. The document percentage alone does not establish that byte total. Common Crawl WET files preserve navigation menus, cookie banners, and footer links as text. Penedo et al. instead re-extracted raw WARC records with trafilatura and deduplicated aggressively. They reported that carefully processed web data could match corpora built from human-selected sources. The same crawl therefore produces a different model when the extractor changes. Without extractor name, version, and thresholds, the training recipe is incomplete and unrecoverable.

The provenance boundary

Raw crawl records can retain a Uniform Resource Locator (URL), timestamp, and license header. Text extraction keeps the URL but can drop license-bearing markup. Filtering retains document objects, so an auditor can still sample and estimate a rate. Tokenization, shuffling, and packing dissolve document identity. Below that point, prompting is an extraction test rather than a lookup. A closed-data model therefore supports behavior measurements, not exact claims about corpus membership. Open weights do not imply open training data. Composition reports for released corpora do not establish the composition of a different closed corpus. The curation-first alternative also fails as a complete answer. Reading a paragraph reveals content but not who owns it, whether the site could post it, or when a license may end. That information is provenance metadata discarded earlier in the funnel. Without a retained key, removing a document changes from a file operation into a retraining problem.

Worked audit economics

Using a planning prevalence $p=0.02$, a nominal $95\%$ normal-approximation interval, and half-width $e=0.005$ (that is, $0.5$ percentage points), the sample-size formula gives $n\approx3{,}011.81$. Round upward to $n=3{,}012$ documents. This assumes approximately independent random sampling from a large corpus and a credible planning value for $p$; it is not a prevalence-free coverage guarantee. If no prevalence estimate is available, the conservative choice $p=0.5$ gives $n=38{,}416$ instead.

At $5$ minutes per document, the $3{,}012$-document audit takes $251$ person-hours and costs $\$3{,}765$ at $\$15$ per hour. With two reviewers each working $40$ hours per week, the elapsed time is $251/(2\times40)=3.1375$ working weeks, not less than one week. This assumes each sampled document receives one review; independent double review would double the labor. Enumeration has a different scale. A reader at $250$ words per minute, assuming about $5.7\,\mathrm{B/word}$ and $4.8\,\mathrm{B/token}$, covers roughly $300$ tokens per minute. Using the explicitly rounded planning rate of $18{,}000$ tokens per person-hour, reading $15.6\times10^{12}$ tokens takes approximately $8.67\times10^8$ person-hours. At $2{,}000$ hours per person-year, that is $4.33\times10^5$ person-years; at $\$15$ per hour, it costs approximately $\$13.0$ billion.

The Llama 3 405-billion-parameter run used $30.84\times10^6$ H100 graphics processing unit (GPU) hours. At $\$2.50$ per GPU-hour, the worked cost is $\$77.1$ million. The $\$13.0$ billion reading estimate is about $169$ times that training cost and $3.45\times10^6$ times the $\$3{,}765$ sampled-audit cost. The audit-to-training cost fraction is approximately $4.88\times10^{-5}$, or $4.88$ parts in $100{,}000$. A $300$-page book is approximated as $100{,}000$ words and $120{,}000$ tokens. The corpus is therefore $15.6\times10^{12}/120{,}000=1.30\times10^8$ book-equivalents. The chapter compares that order with Google’s 2010 count of $129{,}864{,}880$ distinct published books. This is only an order-of-magnitude check, not evidence that the corpus contains that many distinct books.

Practice decisions

Default to a sampled audit with a stated interval. Do not replace the interval with the word “clean.” The source suggests switching toward exact matching with a hash or n-gram index when prevalence falls below roughly $p=0.005$. Treat this as a workflow heuristic, not a universal statistical threshold: sample requirements depend on absolute or relative error targets, confidence, and the detection task. Exact matching also needs reference material. Record the extractor, its version, and all filter thresholds next to the seed and learning rate. Route content that can be revoked in under one year to a datastore by default. The source allows an exception for public-domain and permissively licensed text where revocation risk is treated as zero. Prefer a trained quality classifier to a keyword blocklist, except where a legal prohibition names literal strings. The source reports that C4’s blocklist removed documents about minority identities at a disproportionate rate, so its toxicity benefit also costs long-tail coverage. For a corpus you did not build, report an extraction probe rather than asserting that a filter removed the content. Treat a stated $0.3\%$ training mixture as a mixing weight, not as an extraction rate, because a small duplicated source can be easier to extract than a larger diverse source.

5.2 PII extraction and the divergence attack

Alignment changes behavior, not stored associations

Pre-training fits parameters $\theta$ to a next-token distribution over the corpus. Supervised fine-tuning and preference optimization produce later parameters $\phi$ from a much smaller set of demonstrations. Those updates reshape likely continuations in covered contexts. They are not a selective deletion procedure for a phone number repeated hundreds of times in pre-training. Memorized associations can remain after alignment; refusal behavior is not proof that sensitive data is absent.

Duplication and marginal sampling

If a corpus contains $M$ equally weighted sequence instances and sequence $s$ appears $d(s)$ times, its empirical sequence probability is $\widehat p(s)=d(s)/M$. Maximum-likelihood training favors fitting that distribution. The approximation below is schematic: finite model capacity, conditioning, and optimization prevent an automatic equality between the trained model and the empirical distribution.

An unconditional sample therefore favors heavily duplicated sequences. Regurgitation in this regime is sampling that reflects the corpus distribution. Alignment normally starts sampling inside contexts covered by assistant demonstrations. Its security property belongs to that starting region, not to deletion from weights.

Divergence

A repetition instruction can drive the model outside contexts represented in alignment data. A long repeated context carries little information about the intended topic or task. When the loop breaks, generation may expose pre-training associations, including duplicated text; approaching the unconditional pre-training marginal is a heuristic explanation, not a distributional identity. Nasr et al. measured roughly $150$ times more training-data emission under divergence than during normal behavior. They found memorized PII in $16.9\%$ of tested divergent generations. About $\$200$ in hosted queries recovered more than $10{,}000$ unique verbatim training examples. Carlini et al. sampled from a $1.5$-billion-parameter Generative Pre-trained Transformer 2 (GPT-2) model and hand-checked $1{,}800$ candidates. They confirmed $604$ verbatim training examples, giving $604/1{,}800\approx33.6\%$ precision. One example combined a person’s name, email address, phone number, fax number, and physical address. The chapter cites General Data Protection Regulation (GDPR) Article 4(1) for the definition of personal data as information relating to an identifiable natural person. A name beside an employer and city can qualify under that definition even when no regular expression marks it. This chapter preserves that legal statement as reported and does not extend it into case-specific advice.

Why the common defenses lose

Blocking the literal repeat-forever string patches one prompt, not every way to reach an off-support context. The limitation is coverage and generalization, not a contrast between an “unbounded” prompt space and an “enumerable” blocklist: bounded-context prompt spaces are finite, and unrestricted finite token strings are themselves enumerable. Output redaction also faces an attacker making many attempts, who may need only one failure. It recognizes only a subset of information that can identify a person. These controls do not by themselves erase the underlying associations. A divergence attack is not indirect prompt injection. Injection uses adversarial text retrieved from a datastore; divergence need not touch the index, so retrieval-side controls alone do not stop it.

Worked probe economics

The worked example uses a generator with $N=7\times10^9$ parameters, fine-tuned on $200{,}000$ support transcripts. It applies the forward-pass proxy $2N=1.4\times10^{10}\,\mathrm{FLOP/token}$ to both prompt and completion tokens, with the common throughput and price assumptions above. Combining the reported $16.9\%$ PII-bearing-generation rate with the $150$-fold training-data-emission multiplier gives a constructed baseline of $0.169/150\approx0.00112667$, or $0.112667\%$. This combination assumes the multiplier transfers to that PII metric; it is not a directly measured normal-prompt rate. For $10{,}000$ normal queries with $100$ prompt and $200$ completion tokens, the total is $3.00\times10^6$ tokens. The idealized estimate is $123.53\,\mathrm{s}$, $0.03431$ GPU-hours, and $\$0.08578$. The expected count is $11.27$ PII-bearing generations, giving about $\$0.00761$ per such generation. For $10{,}000$ divergence queries with $50$ prompt and $2{,}000$ completion tokens, the total is $20.5\times10^6$ tokens, giving $844.12\,\mathrm{s}$, $0.23448$ GPU-hours, and $\$0.58619$.

At the assumed $16.9\%$ rate, the divergence probe has an expected $1{,}690$ PII-bearing generations and an idealized cost of approximately $\$3.47\times10^{-4}$ per PII-bearing generation. It uses $20.5/3\approx6.83$ times the compute of the normal probe, while the modeled cost per PII-bearing generation is about $21.95$ times lower. These are output counts, not counts of unique records or customers: a generation may identify multiple people, and the same person may recur. The $200{,}000$ training transcripts do not supply a distinct-customer denominator, so a customer-exposure percentage cannot be inferred. The hosted attack’s rounded $\$0.02$ per unique training example is numerically about $57.7$ times the local cost per PII-bearing generation, but those are different outcome units. The local proxy is approximately $\$0.0286$ per million processed tokens; comparing it with the cited $\$2.00$ per million hosted output tokens gives a numerical ratio of about $69.9$. Prompt versus output billing, unique-example deduplication, different target models, and omitted serving overhead prevent these ratios from validating each other.

Practice decisions

Use a divergence probe as a release gate. The source’s default is $10{,}000$ repetition prompts at the production completion limit. On the worked $7$-billion-parameter model, the FLOP proxy gives about $\$0.59$ and $14.1$ minutes on one GPU; actual latency and cost must be benchmarked. For a third-party Application Programming Interface (API), use a smaller $500$-query sample when flooding is not possible and label extrapolation as extrapolation. Do not fine-tune the generator on records subject to erasure requests. Use synthetic or consented transcripts for tone when that satisfies the product goal. Treat output PII redaction as a detector, not as the control that proves release safety. The source permits a narrower exception for checksum-verifiable fields such as card numbers. Limit completion length to product need because divergence needs a long run. The source’s suggested relationship between completion caps and attacker cost is a hypothesis to measure, not a universal linear law. On long-form surfaces, stop repeated n-grams before the model clears the repetition phase. Report PII-bearing generations per dollar separately from deduplicated records or people per dollar. Route a confirmed finding to the weights-datastore boundary rather than treating more guardrails as a durable repair.

5.3 Copyright, Books3, and verbatim continuation

What the evidence can and cannot show

Books3 contains roughly $196{,}000$ English-language books. It was assembled in 2020 from a shadow library and released as about $101\,\mathrm{GiB}$ of plain text. The Pile included it as roughly $12\%$ of $825\,\mathrm{GiB}$. The chapter reports that Books3 was named in the complaint in Kadrey v. Meta Platforms, filed in the Northern District of California in July 2023. It also reports that the Danish Rights Alliance had the dataset removed from public distribution in 2023. That removal did not change models already trained on the dataset. Given a canonical book, a team can prefix the model with $k$ tokens and measure its longest exact continuation. A long verbatim continuation is strong evidence of training exposure, but the chapter does not call it proof of corpus membership. The opening legal exhibit is a $400$-word exact continuation from a novel. Without that calibration, an engineering measurement can be overstated in a legal setting.

Independent errors versus lock-on

The naive model assigns independent probability $p$ to each correct next token, conditional on a fixed supplied prefix. The probability that the next $L$ tokens are all correct is then $p^L$. With $p=0.95$, the $100$-token probability is approximately $5.92053\times10^{-3}$, and the $300$-token probability is approximately $2.07530\times10^{-7}$. These are small probabilities, not impossibility claims; the chance of at least one success also depends on the number of attempts.

A long exhibit motivates questioning the independence assumption; one observed run alone does not mathematically refute it without a sampling model and attempt count. Each correct token can give more evidence that the context is a specific document. Induction heads can match a token against earlier context and copy what followed it. The schematic model assigns lock-on probability $\pi(d,k)$, increasing with duplication count $d$ and supplied prefix length $k$. Conditional on lock-on, take per-token accuracy $q=0.999$. Then the probability of $300$ further correct tokens is $q^{300}\approx0.740707$.

About three quarters of already locked-on generations therefore run $300$ further tokens clean in this toy model. A two-regime survival model makes the conditioning explicit:

The $0.740707$ value is not the unconditional extraction probability; it must be weighted by $\pi(d,k)$, with the non-locked contribution also included. A mixture can create an initially steep decay followed by a much slower tail. It does not automatically prove bimodality, and for $p,q\lt 1$ both terms eventually approach zero rather than a permanent nonzero plateau. Mean overlap can obscure a rare long exact run. Track the maximum run for tail exposure, but compare maxima only at matched attempt counts and generation caps.

Output filters and reported legal holdings

An n-gram filter can block literal matches to indexed works. The chapter cites a roughly $150$-character public-code match window documented for GitHub Copilot. That filter misses works not present in its index. It also does not test non-literal copying. The chapter states that substantial similarity can reach structure, sequence, and expression beyond exact text. A clean report should therefore say “no literal matches above $150$ characters against the works we indexed.” It should not be restated as “no infringement.” The chapter reports that in Bartz v. Anthropic, in June 2025, Judge Alsup held training on lawfully acquired books transformative fair use and separately held that downloading and retaining pirated copies for the training library was not. It reports that the case settled in 2025 for roughly $\$1.5$ billion. It reports that Kadrey granted summary judgment to Meta on the record those plaintiffs built while stating that the ruling did not establish that training on pirated books is lawful. These are the chapter’s descriptions of specific proceedings, not a general legal rule or present legal advice. An output filter cannot reach an acquisition act that occurred before training.

Three Books3 configurations

The worked example assumes $196{,}640$ books, $80{,}000$ words per book, and $1.3$ tokens per word, giving $104{,}000$ tokens per book and $20{,}450{,}560{,}000$ tokens in total. Dividing the reported $\$1.5$ billion settlement by an assumed $500{,}000$ works gives $\$3{,}000$ per work. Applying that scenario rate to Books3 gives $\$589.92$ million. At $\$15$ per ebook, the purchase-price scenario is $\$2{,}949{,}600$, and the ratio is exactly $200{:}1$ under these assumptions. This is a sensitivity comparison, not an estimate of actual liability or a claim that buying an ebook grants training rights. For a test-time datastore, use non-overlapping $512$-token chunks and do not let a chunk cross a book boundary. Retaining the final partial chunk gives $\lceil104{,}000/512\rceil=204$ vectors per book and $196{,}640\times204=40{,}114{,}560$ vectors overall. At $768$ dimensions in fp32, each vector uses $768\times4=3{,}072\,\mathrm{B}$. The vector payload is therefore $123{,}231{,}928{,}320\,\mathrm{B}\approx123.23\,\mathrm{GB}$, excluding source text, graph links, identifiers, and metadata. Embedding the unpadded source tokens at $\$0.02$ per million tokens costs about $\$409.01$. Removing one modeled title selects $204$ vectors. The datastore does not make acquisition lawful by itself; it changes the mechanics of a later opt-out. The chapter reports United States Copyright Act Section 504(c) statutory damages of $\$750$ to $\$30{,}000$ per work, rising to $\$150{,}000$ for willful infringement. The $\$3{,}000$ scenario rate is $4$ times the stated floor and $2\%$ of the cited willful ceiling. Multiplying $196{,}640$ by $\$150{,}000$ gives $\$29.496$ billion as an arithmetic upper-sensitivity scenario, not a predicted award; eligibility and legal findings are separate questions.

Practice decisions

Require acquisition provenance before accepting a training corpus. Use a receipt, license, or public-domain designation per source. Probe at production prefix length and report the maximum exact run alongside the number of attempts and the completion cap. The source suggests drawing a few thousand prefixes from works that can be named. Use a high percentile for a stable trend only when the strict maximum remains the gate. Ship an n-gram filter as a symptom control, not as the complete risk control. Deduplicate before reducing model size because the chapter cites roughly a $10$-fold reduction in memorized emission after deduplication with no held-out perplexity cost. Route revocable content to a datastore before training. Record a retirement date for checkpoints trained on data with unproven provenance. When a retrieval-only license costs $4$ times a training-inclusive license, price revocation, termination notice, and retraining before treating the cheaper license as lower cost.

5.4 Unlearning and why it cannot scale

Exact and approximate objectives

Let $\mathcal D_{\mathrm{train}}$ be the training corpus, $F$ the true training sequences to forget, and $R=\mathcal D_{\mathrm{train}}\setminus F$ the retained corpus. Exact unlearning targets the same model-output distribution as training on $R$ with the specified training procedure and randomness. The chapter’s baseline obtains this by retraining on $R$; exactness is the equivalence requirement, not a theorem that every possible exact method must rerun all training. This guarantee requires correctly identifying and excluding $F$ and all relevant prior exposure.

where $\xi$ denotes the training randomness. An exact unlearning procedure should match the distribution of $\mathop{\mathrm{Train}}\nolimits(R;\xi)$ under the same prescribed randomness law. Bourtoule et al. made exact deletion cheaper for classifiers by sharding data and retraining one shard. For the chapter’s $7$-billion-parameter generator, the full-retraining baseline costs approximately $\$1.338$ million and $22.30$ days on $1{,}000$ GPUs under ideal scaling. Approximate unlearning instead tries to lower probability on the observed candidate forget set $\widehat F$ while replaying retained examples $R'$. Let $\beta\ge0$ weight forgetting against retention; $\beta$ is distinct from the retrieval-mixture weight $\lambda$ used later. One normalized surrogate objective is

This surrogate encourages suppression of supplied examples but does not guarantee that every example’s probability falls. Without constraints or regularization, the negative log-probability term can make the objective unbounded above as forget probabilities approach zero; practical methods need safeguards such as regularization and early stopping. The worked update-plus-evaluation compute estimate is about $\$0.663$ per request. Enumeration and interference remain separate problems even if that compute is cheap.

Enumeration failure

An erasure request names a person, book, or customer; the optimizer consumes token sequences. If document-to-token provenance was discarded during tokenization and packing, $\widehat F$ contains only candidates surfaced by finite probes. Recall against the true set is $\lvert\widehat F\cap F\rvert/\lvert F\rvert$ for $\lvert F\rvert\gt 0$, and cannot be measured without knowing $F$. A candidate also need not be a verified member of $F$. Re-running the same prompts after optimization tests suppression on those prompts, not absence under untried prompts. Benchmarks can make a stronger claim because they supply the forget set by construction. The chapter cites a fictitious-author benchmark and a curated hazardous-knowledge benchmark as legitimate optimizer tests that do not measure production enumeration. It also cites evaluations where reported forgetting reappears under adjacent-data relearning or another language.

Interference failure

Consider the forget-only component of an update. With $g_{\widehat F}$ defined as the gradient of the mean forget-set log-likelihood, suppressing that likelihood requires a negative step, $\theta'=\theta-\eta g_{\widehat F}$ for $\eta\gt 0$. A positive step would increase forget-set likelihood to first order. For a retained sequence $y$, the first-order change is controlled by the gradient inner product. It vanishes when the gradients are orthogonal, but a zero first-order term does not rule out higher-order changes. The full surrogate update also includes the replay gradient.

Shared parameters need not provide isolated directions for deleting individual facts. Applying the chapter’s empirical estimate of $3.6$ memorized bits per parameter to $7\times10^9$ parameters gives $3.15\times10^9\,\mathrm{B}=3.15\,\mathrm{GB}$, or about $3.2\,\mathrm{GB}$. This is an extrapolated memorized-information estimate, not the physical checkpoint size or a universal model-capacity bound.

The $75\,\mathrm{TB}$ corpus is about $75{,}000/3.15\approx23{,}810$ times this memorized-information estimate. Corpus bytes and memorized information are different quantities, so this ratio alone does not prove a particular interference rate or storage mechanism. Shared representations and superposition explain why a forgetting update can change retained behavior in either direction. Replay can reduce damage but cannot guarantee orthogonality to every retained gradient.

Stream compounding

Under the illustrative assumption that each run loses the same fraction $\delta$ of the remaining retained capability, with no recovery or other changes, the normalized capability after $n$ runs is $(1-\delta)^n$. More generally it is $\prod_{i=1}^n(1-\delta_i)$. These are scenario models, not measured laws of unlearning.

At $\delta=1\%$ and $n=400$, normalized retention is $0.99^{400}\approx1.795\%$. At $\delta=0.1\%$, it is $0.999^{400}\approx67.019\%$. Retaining at least $95\%$ after $400$ runs requires the exact bound $\delta\le1-0.95^{1/400}$, approximately $1.28225\times10^{-4}$ as a fraction, or $0.0128225\%$ of remaining capability per run. That is approximately $1.28225$ parts in $10{,}000$. Measurement precision depends on the metric and test design, as calculated below. At $12$ requests per month, $n=144$ in one year; the same $1\%$ relative-loss scenario leaves $0.99^{144}\approx23.517\%$. The system pays for a request stream, not one apparently cheap request.

Why sharding loses for generation

With $S=100$ separately served shard models of $N=7\times10^9$ parameters each, the system carries $SN=7\times10^{11}$ parameters. At fp16, parameter storage is $1.4\,\mathrm{TB}$ rather than $14\,\mathrm{GB}$ for one model, excluding activations, caches, and other overhead. If every query evaluates all $S$ models and combines their predictions, inference work is approximately $100$ times a single forward pass; this is an all-shard-ensemble assumption, not an unavoidable cost of every routed or sharded design. Each shard is assigned roughly $1/S$ of the training data, not necessarily $1/S$ of distinct facts. This baseline makes occasional deletion cheaper by adding recurring serving cost. A datastore keeps keyed row deletion without this model-ensemble multiplier.

Worked erasure economics

The example contains $200{,}000$ support transcripts. One customer appears in $40$ transcripts of $500$ tokens each, giving $20{,}000$ forget tokens. Take $400$ annual erasure requests as an independent workload assumption. A $0.2\%$ annual customer rate would imply $400$ requests only with a separately established population of $200{,}000$ distinct customers; a transcript count does not establish that population. Training uses $6N=4.2\times10^{10}\,\mathrm{FLOP/token}$. The full-retraining baseline processes $15.6\times10^{12}$ tokens, requiring about $535{,}294$ GPU-hours and $\$1.338$ million. On $1{,}000$ ideally scaling GPUs, one run takes about $22.30$ days. Four hundred sequential runs take about $8{,}922$ days, or $24.4$ years. Approximate unlearning uses a replay set $100$ times the $20{,}000$-token forget set. Three epochs process $6.06\times10^6$ tokens. The idealized update takes $748.59\,\mathrm{s}$ and costs about $\$0.51985$. A $5{,}000$-question evaluation with $1{,}000$ processed tokens per question takes $205.88\,\mathrm{s}$ and costs about $\$0.14297$ under the forward-pass proxy.

Thus the total is about $\$0.663$ per request and $\$265$ for $400$ requests when calculated before rounding. Full retraining is about $2.02\times10^6$ times more expensive per request under this compute model. To compare evaluation uncertainty with the relative-loss budget, the metric must be specified. Suppose retained capability means accuracy normalized to a known baseline $a_0=0.5$, and evaluation items are independent Bernoulli trials. A single $5{,}000$-item accuracy estimate has nominal $95\%$ half-width $h\approx1.386$ percentage points. The allowable absolute accuracy loss for one run is $a_0\delta_{\max}\approx0.006411$ percentage points, not $\delta_{\max}$ percentage points. The illustrative half-width is therefore about $216.2$ times that absolute budget.

Under those assumptions, targeting that half-width needs approximately $233.65$ million items, or $46{,}730$ times the original sample. This is a confidence-interval illustration, not a universal evaluation lower bound or a powered regression test: paired comparisons, discordance rates, correlated questions, baseline uncertainty, and a different capability metric change the requirement. For a cost cross-check, the source reports $1.3$ million GPU-hours for Llama 3 8B, or $\$3.25$ million at the same hourly rate. Scaling the worked $7$B estimate linearly to $8$B gives about $\$1.529$ million, a ratio of roughly $2.13$. The source attributes the discrepancy to assumed versus implied model FLOPs utilization, about $34\%$ versus $16\%$, and bandwidth limits for small models; the cost ratio alone does not establish that explanation.

Practice decisions

Answer erasure requests against datastores and logs when those are the serving systems that hold the records. The chapter cites the one-calendar-month GDPR Article 12(3) response window in support of an operational deletion workflow; this is not identical to a fixed $30$ days and should not be interpreted here as a universal, exception-free deletion deadline. It recommends refusing an unverifiable promise of removal from model weights. The source proposes a narrower operational promise: delete from systems that serve the customer’s data and exclude the data from later training within $30$ days. That proposed language is the source’s framing, not legal advice from this chapter file. Batch requests into one approximate-unlearning run when such a run is required, while measuring the resulting batch’s damage rather than assuming it equals the damage from one small request. Use an independent team to build the verification probe. Report evaluation uncertainty and test design beside any claim of no detected regression. Protect $\widehat F$ as sensitive data because it is a concentrated list of target sequences.

5.5 SILO: low-risk weights, high-risk datastore

Two memories and one boundary

SILO combines parametric weights $\theta$ and a non-parametric datastore $D$. Let $x$ be the request and $\mathop{\mathrm{RET}}\nolimits(x,D)$ the retrieved context. A generated output $Y$ is sampled from the conditional distribution

Both sides may require a copy of source text. The architecture does not claim that making the datastore copy is lawful. The difference is what the system can do after the copy exists. $D$ can enumerate rows, select them by key, delete them, and check that the serving path excludes them. Weights do not expose those exact document-level operations. The source therefore partitions data by license before any gradient step. Public-domain, permissive-software, and attribution-license tiers may train the parametric model under the source’s policy. Copyrighted, private, or otherwise encumbered content stays in a datastore read at inference. Min et al.’s Open License Corpus has $228\times10^9$ tokens across the three low-risk tiers. They trained $1.3$-billion-parameter models cumulatively on $60\times10^9$, $250\times10^9$, and $350\times10^9$ tokens for the PD, PD+SW, and PD+SW+BY configurations. Cumulative processed tokens can exceed unique corpus tokens through repeated sampling; those are different counts. Without a license field at ingestion, the later removal operation loses its key.

The measured quality tax

On OpenWebText2, reported parametric-only perplexity is $37.8$ for PD, $21.1$ for PD+SW, and $18.8$ for PD+SW+BY. The compute-matched Pythia-1.4B comparison trained on The Pile reaches $11.5$. The strongest low-risk model therefore has perplexity ratio $18.8/11.5\approx1.63478$. Since perplexity is the exponential of mean token log loss, the corresponding loss gap is $\Delta\mathcal L=\ln(18.8/11.5)\approx0.49151$ nats per token.

The source attributes the compute-matched quality gap primarily to corpus composition; matched compute alone does not rule out every other experimental difference. If the reported $90\%$ gap closure is applied to this specific gap in log-loss space, the remaining loss gap is $0.1\Delta\mathcal L\approx0.04915$ nats per token, and the implied perplexity is approximately $12.08$:

Thus the implied residual perplexity premium is about $5.04\%$, versus $63.48\%$ for the parametric-only ratio. This is a conditional derivation, not a separately reported OpenWebText2 measurement, and it is not a $90\%$ reduction in perplexity itself. The benefit is not monotonic across every task: adding BY changes OpenWebText2 from $21.1$ to $18.8$ but Amazon from $34.8$ to $37.0$.

How the datastore changes removal

kNN-LM mixes the parametric distribution $p_\theta(y\mid x)$ with a normalized neighbor distribution $p_{\mathrm{kNN}}(y\mid x,D)$. Here $\lambda\in[0,1]$ is explicitly the weight on the parametric term; some references use the opposite convention.

Setting $\lambda=1$ removes the datastore contribution exactly for that computation. Deleting rows changes the neighbor set on the next query, provided all reads enforce the deletion state. If no training or distillation used those rows, their text was not integrated into the weights through that path. Removal is addressable as a key query, such as all rows for one publisher. Verification should check the selected keys and the read path; one unsuccessful retrieval query alone does not prove universal absence. This structural gain is distinct from suppressing a finite list of model probes. The split does not settle fair use or make retrieval indexing categorically lawful. The chapter states that no court has held indexing for retrieval categorically fair use. It frames SILO as management of a risk-performance trade-off, not elimination of risk.

Treatise collection configurations

The worked collection has $2\times10^6$ pages at $500$ tokens per page, totaling $10^9$ tokens, divided approximately equally among three publishers. Each license has a contractual $30$-day revocation clause, and the system receives $10^6$ queries per month. Using the chapter’s full-retraining baseline for one publisher gives approximately $\$1.338$ million and $22.30$ days, leaving $30-22.3039\approx7.70$ days of idealized schedule margin. That compute baseline assumes a full rebuild of the large training corpus, not merely training over the $10^9$ treatise tokens. Rebuilding from a known retained corpus is a procedural guarantee; tests of the resulting weights alone do not reconstruct historical membership. RIC-LM uses non-overlapping $250$-token chunks, giving $4\times10^6$ vectors. At $768$ dimensions in fp16, each vector uses $1{,}536\,\mathrm{B}$, so the vector payload is $6.144\,\mathrm{GB}$. Under the common HNSW base-layer allocation convention, $M=32$ gives up to $2M=64$ neighbor slots of $4\,\mathrm{B}$ each. This budgets $256\,\mathrm{B/vector}$, or $1.024\,\mathrm{GB}$ for base-layer links. The combined payload-plus-base-layer estimate is therefore $7.168\,\mathrm{GB}$, not a complete all-in index size: upper layers, identifiers, source chunks, allocation overhead, and metadata are omitted. One publisher accounts for approximately $4\times10^6/3\approx1.33\times10^6$ rows; exact integer counts depend on the partition. A $4$-million-bit tombstone bitmap uses $4\times10^6/8=500{,}000\,\mathrm{B}=500\,\mathrm{kB}$. Read-time checks can block revoked rows while rebuilding later. This is logical serving exclusion, not necessarily complete erasure of replicas, backups, logs, or caches. kNN-LM stores one vector per token at $2{,}048$ dimensions. At fp16, the vector payload alone is $10^9\times2{,}048\times2=4.096\times10^{12}\,\mathrm{B}=4.096\,\mathrm{TB}$. Compared with the RIC-LM payload-plus-base-layer budget, the ratio is $4{,}096/7.168\approx571.4$. These are explicitly scoped storage estimates, not all-in serving-memory measurements. The source reports stronger quality and better datastore-size scaling for kNN-LM. It also notes that kNN-LM requires control of the token-probability serving path rather than only a hosted text-generation interface.

Retrieval serving cost

RIC-LM retrieves $k=5$ chunks of $250$ tokens each, adding $1{,}250$ prompt tokens per query. The FLOP proxy gives approximately $0.05147\,\mathrm{s}$ of extra prefill work and $\$3.574\times10^{-5}$ per query. At $10^6$ queries per month, this is about $\$35.74$ monthly. It is only the incremental generation-prefill estimate, excluding search, embedding, storage, fixed serving capacity, and other operating costs.

The $37{,}440$-month comparison uses unrounded costs on both sides; it is an illustrative ratio, not a forecast of stable prices for millennia. The source cross-checks the $4.096\,\mathrm{TB}$ vector payload against a published $10^9$-vector, $2{,}048$-dimension datastore. It also reports a $1.4\times10^{12}$-token datastore where quality improved with datastore size and a smaller model plus a large datastore beat a larger model alone on knowledge-intensive tasks.

Practice decisions

Stamp every document with PD, SW, BY, or high-risk at ingestion. Carry that field onto every chunk and training shard. The source defaults to RIC-LM for hosted APIs; the worked storage comparison is about $571$-fold under the stated budgets, not a universal compression ratio. Use kNN-LM when domain shift makes quality the binding constraint and you own the serving stack. Soft-delete at read time, then rebuild on the normal schedule. The source suggests an immediate rebuild when the removed tier exceeds roughly $1/5$ of the index and per-publisher ablation when one source exceeds $1/4$ of retrieved context. These are workload heuristics to validate, not mathematical recall guarantees. Do not fine-tune on high-risk text or synthetic paraphrases distilled from it under the source’s policy. The source characterizes a generated paraphrase of a licensed treatise as a derivative work that crosses the boundary without the original provenance. The chapter treats content owned outright as an exception. For a tenant-specific quality gain, a separate LoRA adapter can remain deletable if it is never merged, never served across tenants, and contains the only gradients from that tenant. The interview scenario assigns that adapter an $8$-percentage-point exact-match gain, not an $8\%$ relative gain, then makes deletion rather than quality the deciding requirement. Merging the adapter destroys that file-level deletion guarantee. Name residual risk in a contract rather than promising unverifiable removal from weights.

Diagrams

The manifest records five figures and zero source tables. All five figures appear below.

Figure 5.1

Diagram description and source

flowchart TD
    A["Raw WARC crawl: URL, timestamp, and source metadata"]
    B["Text extraction: WET or trafilatura; some provenance is lost"]
    C["Filtering and deduplication: documents remain addressable"]
    D["Tokenize, shuffle, and pack without a retained document map"]
    E[["Model weights: no native document-key lookup or delete"]]
    A --> B
    B --> C
    C -.->|"Provenance boundary in this pipeline"| D
    D --> E

Figure 5.1: In this pipeline, discarding the document map changes a keyed removal into a model-rebuilding problem. The reported C4 funnel is $20\,\mathrm{TB}\to750\,\mathrm{GB}$, retaining $3.75\%$ of bytes and yielding approximately $1.56\times10^{11}$ tokens. Byte and token figures are as reported by Raffel et al. (2020). Tokenization does not inherently prohibit retaining an external provenance map; the boundary is the decision to discard it.

Figure 5.2

Figure 5.2: This is a conceptual context schematic, not a literal geometric fence or a proof that off-support generation equals the pre-training distribution. $\mathcal C_{\mathrm{align}}$ denotes contexts represented by alignment data; both ordinary and repetition prompts are evaluated by the later model $p_\phi$. Duplication count $d(s)$ affects a sequence’s empirical frequency. Alignment is not a certified deletion operation. Nasr et al. (2023) measured roughly $150$-fold higher training-data emission in their divergence setting.

Figure 5.3

Figure 5.3: The independence and two-regime mixture models use $p=0.95$ and $q=0.999$. Without a measured $\pi(d,k)$, an unconditional plateau height or a fixed vertical gap cannot be assigned. For fixed $\pi\gt 0$ and $q\gt p$, the ratio of the two survival probabilities grows with $L$, but their absolute difference does not grow without bound: both probabilities approach zero when $p,q\lt 1$. The independence model predicts rare long runs, not impossible exhibits. These equations are schematic, not a measured run-length curve.

Figure 5.4

Panel a:

Panel b:

Figure 5.4: (a) The obligation concerns $F$, while the optimizer receives the probed candidate set $\widehat F$; its recall cannot be established without knowing $F$. The corpus-byte and memorized-information figures are not directly comparable storage capacities and do not prove a particular interference rate. (b) Under constant relative damage and no recovery, $400$ runs at $\delta=1\%$ retain approximately $1.795\%$ of baseline capability; at $\delta=0.1\%$, approximately $67.019\%$ remains. The exact per-run bound for a $95\%$ floor is $1-0.95^{1/400}$, approximately $1.28225\times10^{-4}$.

Figure 5.5

Panel a:

Diagram description and source

flowchart LR
    PD["Public domain, PD"] --> W["Weights: gradient-based storage without a native document delete"]
    SW["Permissive software, SW"] --> W
    BY["Attribution licenses, BY"] --> W
    HR[["Copyrighted, private, or encumbered"]] --> D[["Datastore: keyed chunks used at inference"]]
    HR -.->|"Source policy blocks this gradient path"| X["No training or distillation on this tier"]

Panel b:

The contractual $30$-day window leaves approximately $7.70$ days after the idealized full retrain. A keyed read-time block can be much faster, but the illustrative $1\,\mathrm{s}$ is not a measured bound or a claim that all copies, caches, replicas, and backups are erased in that time. Figure 5.5: (a) Under the source’s policy, the three low-risk tiers may reach gradients while high-risk material remains keyed in datastore $D$. (b) The comparison is between an illustrative logical serving block and the chapter’s full-retraining baseline, approximately $\$1.338$ million and $22.30$ days on $1{,}000$ ideally scaling GPUs. These are different operational remedies. A rebuild from a correctly identified retained corpus has a procedural input guarantee; probes of the weights alone do not establish corpus membership. The architecture preserves a removal handle but does not settle the lawfulness of the original copy. The source chapter contains no tables.

Whiteboard pack

What to draw

1. Draw a vertical pipeline with crawl, extract, filter, tokenize, and weights.
2. Add a dashed line between filtering and tokenization. Label it “provenance boundary.”
3. Write “file delete” above the line and “retrain” below it.
4. Draw a small aligned region inside a larger weight space. Add a repetition prompt leaving the small region.
5. Draw two run-length survival curves. Show rapid independent-error decay and a slower mixture tail, not a permanent nonzero plateau.
6. Label the true forget set $F$ and the probed candidate set $\widehat F$; do not assume every candidate is verified. Add the scenario formula $(1-\delta)^n$.
7. Split the final board into weights and datastore. Route PD, SW, and BY to weights, and route high-risk text to the datastore.
8. End with the scoped comparison: an illustrative $1\,\mathrm{s}$ logical block versus approximately $\$1.338$ million and $22.30$ days for a full retrain. Distinguish keyed serving checks from historical membership claims.

Spoken script

Think of the model as two memories. Weights are compressed memory. They are cheap to query, but tokenization erased document identity, so you cannot list or delete one source. Alignment changes behavior without erasing memorized sequences, and approximate unlearning acts only on examples a probe found while risking collateral damage. A datastore is addressable memory. If every chunk keeps a license key, a revocation becomes a filtered write that you can verify. SILO therefore sends low-risk text to gradients and keeps revocable text behind retrieval. That boundary manages reversibility and performance. It does not decide whether the copy was lawful.

Interview traps

1. “Legal asks whether partner content is in a closed model’s corpus. Do you certify that it is absent?”

No, because tokenization erased the membership list and absence is not enumerable from the weights. Offer a sampled audit with a confidence interval or a targeted extraction probe, then state the residual uncertainty. The trade-off is that sampling estimates a rate cheaply while exact matching needs a retained source index.

2. “The aligned assistant refuses PII. Why run a divergence probe?”

Alignment fences ordinary sampling into covered contexts but does not delete associations in the weights. A long repetition prompt can leave that region and expose duplicated training text. The probe costs more tokens than a normal evaluation, but the worked example returns records far more cheaply.

3. “Mean overlap with protected works is low. Is that enough?”

No. A mixture of locked and non-locked continuations can create a slow-decaying tail, so many harmless outputs can hide one long exact continuation. Gate on the maximum run at production prefix length and completion cap, and compare matched numbers of attempts. Use a high percentile for stable trend tracking. An n-gram filter reduces literal emission against indexed works but does not answer acquisition or non-literal similarity questions.

4. “Approximate unlearning costs only $\$0.663$. Why not run it after every request?”

Compute is not the only limit. The optimizer sees $\widehat F$, not a verified enumeration of $F$, and updates can interfere with retained behavior. Under the illustrative constant-relative-loss scenario, $\delta=1\%$ over $400$ runs leaves $(1-0.01)^{400}\approx1.795\%$ of baseline capability. Batching can reduce the number of updates, but its damage must be measured; it does not solve enumeration or establish complete deletion.

5. “When would you not put high-risk content in a datastore?”

The source permits weights for content classified as public domain, permissively licensed, or attribution-licensed when its obligations can be honored and revocation risk is treated as zero. Use a datastore when content can be withdrawn, when provenance must remain selectable, or when per-source ablation matters. The datastore adds retrieval latency, storage, and a copy that still needs its own lawful basis.

Key numbers