Chapter 36: Provenance and Adversarial Robustness

00-START-HERE · 01-READING-ROUTES · 35_source_credibility · 37_latency_cost_and_systems Purpose: separate textual attribution, cryptographic provenance, factual credibility, instruction authorization and confidentiality; audit each probability and cost model; and explain the guarantees that do—and do not—follow from a layered RAG defense.

TL;DR

• Attribution checks whether cited text supports a claim. Provenance validation checks specified origin/integrity assertions. Neither alone proves factual truth, and macro-$F_1$ is not an error percentage.
• C2PA signs structured claims linked to assertions and content bindings. Its history can be a graph, not a mandatory complete capture-to-publication chain. A valid signature, valid asset binding and trusted credential are different checks.
• Treat missing, broken, untrusted, compromised and anomalous credentials separately. Publish the ranking/exclusion policy and recall trade-off; absence is not proof of forgery.
• Retrieved third-party instructions can redirect a RAG system. Role markers and learned hierarchy help, but privilege enforcement, least-privilege tools and authorization must not depend only on the model obeying prose.
• Datastore poisoning is a persistent ingestion attack and can carry indirect instructions or factual misinformation. Tiny corpus prevalence does not bound concentration in a targeted query’s selected evidence.
• Guardrails, red teams and adversarial training are complementary controls, not independent failure probabilities that may automatically be multiplied. Benchmark detection, residual attack success, false positives, utility and latency.
• Continuous and discrete attacks have different spaces and cost units. MixAT applies continuous perturbations around sampled discrete attack inputs; the reported gains remain scoped to the evaluated models and attack suite. Deletability does not imply confidentiality: authorize before any restricted content reaches an unauthorized consumer.

The story

Picture a newsroom with a giant archive. The editor asks a reporter to answer a breaking question. The archivist is the retriever. The reporter is the large language model (LLM). The shelves are the datastore. The newsroom’s written assignment is the system prompt. The archivist first finds a clipping that states the exact fact the editor wants. That match is attribution. It says the clipping’s words support the reporter’s sentence. It does not say the clipping came from the newspaper named at the top. At the loading dock, a provenance inspector checks signed claims and the asset portions they bind. A new editor may attach a manifest that references several ingredients. The records can reveal an authenticated history that exists; they do not guarantee that every earlier edit or the original capture was recorded. Some envelopes have no usable credential. That is an unknown provenance state, not proof of an attack. The inspector follows an explicit policy: require credentials in a closed workflow, or retain separately labeled and independently corroborated legacy evidence when coverage matters. An attacker can strip metadata, but benign processing can do so too. A seal may verify even when its key was compromised or its signer made a false assertion. The inspector checks trusted roots, policy scope, relevant times and available revocation evidence. An unusual edit pattern triggers review, not an automatic conclusion that the signature was forged. A clipping can contain white-on-white text instructing the reporter to favor its author. A text extractor may recover words a reader misses. The model can distinguish roles learned during training, but those role markers are not an operating-system privilege boundary. The application must constrain actions and data access outside the generated prose. The archive can be poisoned before a question arrives. Five deliberately query-matching clippings may dominate a small retrieved window even among millions of honest records. Their local share is a retrieval outcome, not a random sample of the corpus, and their influence is not determined solely by how many slots they occupy. The newsroom therefore adds a security desk. A fast guard checks each incoming request against known jailbreak patterns. A red team keeps inventing new disguises. The training team teaches the reporter to refuse those hard examples. These roles remain separate because each catches a different failure. The training team has two drills. One searches realizable word sequences; the other perturbs a numerical embedding within a declared norm budget. Local continuous search is often cheaper for a chosen configuration, but it includes points no token sequence realizes and can miss a distant textual attack. The hybrid drill samples discrete attack inputs and searches continuous neighborhoods around them. Cached persuasive paraphrases can supply those inputs without a fresh expensive token search at every anchor. Coverage still depends on which attacks, perturbations and utility constraints are trained and tested. Deleting a clipping can stop future authorized serving from the live store if all reads enforce the deletion state. It does not recall prior prompts, replicas, caches or logs. Confidentiality instead requires checking each requester’s current permission before a clipping reaches that requester or an unauthorized model/provider. The newsroom needs distinct checks rather than a single “safe” score. Relevant evidence can be inauthentic, signed evidence can be false, credible text can contain an unauthorized instruction, and a useful record can still be forbidden to this requester. Residual risks remain after all checks.

Decoder table

Core mechanics

Mathematical conventions

A score in $[0,1]$ is not automatically a calibrated probability. Keep attribution, credential-validation status, factual credibility and access authorization distinct. Missing, invalid, untrusted, revoked/compromised and anomalous evidence are different states; a policy may map states to ranking weights without making those weights truth probabilities. Cryptographic validity authenticates a signed statement under specified validation assumptions, not every real-world assertion in it. Use decimal units, with $1\,\mathrm{MB}=10^6\,\mathrm B$, full intermediate precision and rounding only at reporting. Hashing, signature checks, certificate/status work, network retrieval and cache lookups are separate costs. Count signed manifests rather than assertions when estimating signatures; state which historical assets are actually rehashed. Report component service-work budgets separately from end-to-end wall latency. The injection-exposure equation assumes IID passage draws with a stated marginal attack probability. The hypergeometric example instead conditions on exactly five restricted candidates and uniform sampling without replacement. Neither is a model-free law for deterministic nearest-neighbor top-$k$. Corpus prevalence, selected-passage prevalence, attack exposure and successful compromise are different quantities. Expectations are not realized incident counts. Define attack-success denominators, behavior IDs, attack budgets, models and judge versions. ALO-ASR is a union of successes across attacks on the same behaviors, not an average over attacks or a guarantee against all possible attacks. Candidate forward evaluations, gradient steps, batched executions and elapsed time are not interchangeable. Correct authorization yields a no-forbidden-context invariant only when identity, metadata, enforcement and all relevant paths are correct.

36.1 Attribution, provenance and truth are different questions

What it is

Attribution asks whether document $d$ supports claim $c$ under a declared rubric. A human or evaluator may assign $a(c,d)\in[0,1]$, but that score need not be a calibrated support probability. AIS separates interpretability from attribution to identified sources. AttributionBench reports around $80\%$ binary macro-$F_1$ for fine-tuned GPT-3.5; it does not imply one incorrect judgment in five. For two classes with defined denominators, $F_{1,j}=2\mathrm{TP}_j/(2\mathrm{TP}_j+\mathrm{FP}_j+\mathrm{FN}_j)$ and $F_{1,\mathrm{macro}}=(F_{1,1}+F_{1,2})/2$, whereas error is $(\mathrm{FP}+\mathrm{FN})/n$. For example, confusion counts $(\mathrm{TP},\mathrm{TN},\mathrm{FP},\mathrm{FN})=(80,80,20,20)$ and $(135,35,15,15)$ both give macro-$F_1=0.8$ but error rates $20\%$ and $15\%$. The benchmark tests textual attribution, not cryptographically certified source authenticity.[1]

Why it exists

Provenance validation reads signed claims, asset bindings, certificates and policy evidence; attribution judges the relation between claim and text. They answer different questions without being statistically independent or necessarily using disjoint features in every implementation. A fabricated document can closely restate a target claim and receive high $a(c,d)$ while lacking acceptable provenance. Conversely, a validly signed document can be factually wrong or fail to support a cited claim. A URL or quotation alone proves none of these properties.

Failure without it

In a four-candidate Drug X toy, stipulate a real trial report with $a=0.94$ and accepted provenance, and a fabricated press release with $a=0.97$ and excluded provenance. The claimed symptom reduction is $40\%$; no clinical result is independently established here. Attribution-only ranking would prefer $0.97$ over $0.94$, while a correctly implemented provenance gate under this stipulated policy removes the known fabrication. That illustrates separate failure modes, not a theorem that all attribution-only systems prefer fakes or that the gate detects every fake. AttributionBench’s textual score does not establish a security bound conditional on all documents having verified $p(d)=1$.

Cost, complexity, and decisions

Log attribution, provenance-validation state and factual credibility separately. An attribution score alone cannot distinguish an attacker’s linguistically matching fabrication from genuine supporting text in an otherwise identical textual test, but shared features can correlate with provenance; “no asymptotic power” is not established here. In a closed credentialed workflow, fail closed when required provenance cannot be validated. In a legacy/open corpus, choose an explicit unknown/corroborated/excluded policy and measure recall and abuse costs. Do not label missing evidence forged or silently set every legacy record to zero. First-party ingestion can supply provenance evidence but still needs integrity, compromise and policy checks; it is not a reason to assume factual truth.

36.2 Content credentials and C2PA

What it is

C2PA represents provenance with manifests containing assertions, a claim and a claim signature. The claim references assertion hashes, including a format-specific hard binding to the asset; ingredient assertions can reference other manifests/assets. A manifest can describe multiple actions, so “three assertions” does not imply three signatures. A graph may have multiple ingredients, begin after capture, or omit inaccessible history. The following notation is schematic, not an implementation of the standard’s byte serialization or COSE signing rules.[2]

Here $\mathcal B_i$ denotes the asset portions covered under the chosen format’s hard-binding rules, including required exclusions. Hashing the entire file including its own embedded manifest would be self-referential and is not the general rule. Let $C_i$ denote the correctly constructed signed message/COSE structure carrying the claim; abstractly:

A signature check authenticates the signed structure under its public key, assuming secure algorithms, correct implementation and uncompromised keys. Separately validate asset hard bindings, assertion references, applicable certificate path/trust anchors, credential policy and relevant timestamps/status evidence. A public key alone does not establish a real-world identity or truthful assertion. The browser/TLS analogy is about certificate-path trust, not identical formats, guarantees or runtime.

• Check the asset portions and assertion references required by the relevant binding/format.
• Check the claim signature under the appropriate public key and algorithm.
• Evaluate certificate path, trust scope, relevant time/status evidence and application policy; report each failure separately. Following ingredient references reveals available recorded history, potentially as a directed graph rather than a linear chain $H_{i-1},\ldots,H_0$. It need not reach an original camera capture or prove that all edits were logged. A graph walk has $O(V+E)$ reference/check overhead for $V$ visited records and $E$ links; hashing bytes, fetching assets and verifying signatures add their own size- and implementation-dependent costs. A simple chain is a special case.

Why it exists

A changed covered asset portion can fail the hard-binding check even while the unchanged manifest signature remains cryptographically valid. An untrusted signer can also produce a valid signature; trust rejection is not signature failure. Modifying signed claims without an authorized signature should be detected under the cryptographic assumptions, but a trusted dishonest or compromised signer can sign new false claims. C2PA provides tamper evidence for recorded bindings and statements, not prevention of all edits, complete historical observation or a truth certificate. A signed AI-edit assertion records what the signer asserted, not independently established ground truth.[2]

Failure without it

Ordinary editable EXIF and a painted visible watermark alone do not provide authenticated structured history. C2PA distinguishes hard cryptographic bindings from soft bindings such as fingerprints or invisible watermarks. Changes to covered content generally invalidate an old hard binding; the standard does not say every byte change anywhere must fail. A soft binding may help recover a separated manifest or identify a rendition, but recovery does not make the original hard binding valid for different bytes. Remote storage alone does not make credentials survive recompression: verify a supported rendition relationship/new signed manifest or report the limits of recovered provenance.[2]

Cost, complexity, and decisions

For an explicitly component-only toy, ingest $10{,}000$ images per day, each $3\,\mathrm{MB}$, with three signed manifests (capture plus two edit records), not merely three assertions. Assume current-asset hashing throughput $500\,\mathrm{MB/s}$ and signature verification $0.3\,\mathrm{ms}$ each; these are stipulated service rates, not a reproduced OpenSSL benchmark. Hashing only the current asset costs $6\,\mathrm{ms}$; three signature checks add $0.9\,\mathrm{ms}$, totaling $6.9\,\mathrm{ms}$ per item or $69\,\mathrm s$ of serial work per day. Checking twenty items serially costs $138\,\mathrm{ms}$, $69\%$ of a $200\,\mathrm{ms}$ budget. The coarser planning rate $7\,\mathrm{ms}$ instead gives $70\,\mathrm s$, $140\,\mathrm{ms}$ and $70\%$; these are rounded planning figures, not separate measurements. This ledger omits certificate/path/status work, manifest/assertion hashing, parsing, I/O and historical asset rehashing. If all three asset states were separately available at $3\,\mathrm{MB}$ each and rehashed, the hash-plus-signature component would be $18.9\,\mathrm{ms}$ per item, $189\,\mathrm s$ daily and $378\,\mathrm{ms}$ for twenty serial items. Historical data may be unavailable; signature verification alone does not establish a full asset-by-asset capture history. Parallelism can lower wall time but not erase total service work or contention. Verify immutable content/signature bindings at ingestion and cache by the appropriate asset/manifest identifiers. Keep mutable trust/status policy separate, version the verifier and re-evaluate on trust changes. Discover embedded or externally stored manifests and available soft-binding recovery without assuming re-encoding preserved the hard binding. Walk the needed ingredient graph for high-stakes use and explicitly report missing history. Choose public or private trust anchors for the deployment’s signer policy; neither choice guarantees factual reliability or a fixed recall/latency.

36.3 Missing, invalid and compromised credentials

What it is

Distinguish no discovered credential, unavailable external data, hard-binding mismatch, invalid signature, untrusted signer, time/status-policy rejection and a merely anomalous history. A valid signature on a dishonest statement or one made with a compromised key is not a cryptographic signature forgery. Let $p(d)$ be a declared provenance-policy weight, not a probability that the document is true. A policy may assign accepted evidence weight one, excluded evidence zero and unknown/corroborated evidence intermediate values. Its threshold does not transform an anomaly into proof of fraud.

Why it exists

Metadata stripping can avoid a credential check, but benign workflows and legacy material can also lack manifests. Whether to exclude or down-weight unknown provenance depends on the required workflow and the measured false-positive/recall trade-off. Trusted-root scoping limits accepted signer credentials; it does not establish honest reporting. Evaluate revocation/status against the standard’s applicable signing-time/timestamp and trust rules: a credential revoked later need not make every historical signed record invalid. An application may also quarantine a known compromised signer under a separate current-risk policy. The example of forty-seven edits across three sessions is a review heuristic, not a calibrated authenticity test.[2]

Failure without it

Neither missing metadata nor an unusual edit count proves forgery. A professional editor may trigger the anomaly heuristic, and an attacker may avoid it. Keep cryptographic failures, status-policy exclusions and review flags separate in logs and user explanations. Do not blocklist a signer merely because one benign history looked unusual; use independently validated compromise evidence and an appeal/review policy.

Cost, complexity, and decisions

Stipulate $N=200$ candidate images partitioned into $120$ accepted, $50$ with no discovered manifest, $18$ with hard-binding mismatches consistent with possible re-encoding, and $12$ requiring status/anomaly review. The counts sum to $200$; neither the eighteen nor the twelve are automatically proven fraudulent. For this ranking toy only, require $u(d)\in[0,1]$ after a declared normalization and use nonnegative $p(d)\in[0,1]$. A raw unbounded dot product or signed cosine score would not justify the following cap. If accepted documents have $p=1$ and $u\ge u_{\mathrm{floor}}=0.3$, then $p_{\min}\lt 0.3$ keeps any unknown item’s weighted score strictly below any such accepted item. Choose $p_{\min}=0.2$. Excluding unknowns from an entire top-twenty list additionally requires at least twenty eligible accepted comparators or a hard exclusion rule. As a stipulated corroboration policy, assign $p_{\mathrm{corr}}=2/3$ when two of three independently assessed trusted sources support the relevant claim; that fraction is not a calibrated asset-authenticity probability. Let the twentieth accepted comparator have $u=0.55$. Use:

An unknown item with $u=0.95$ gets $0.19$. A corroborated item with $u=0.85$ gets $0.85(2/3)=17/30\approx0.566666667$, above the stipulated $0.55$ comparator; do not compute with the rounded $0.67$ as if exact. This can displace that comparator, but final membership also depends on all other competing items. An explicitly excluded item gets zero; a tied zero score still needs a removal/tie policy if zero means “never serve.” For a deliberately serial, unshared network scenario, twenty status requests at $30\,\mathrm{ms}$ each total $600\,\mathrm{ms}$, three times the $200\,\mathrm{ms}$ budget. Certificate sharing, stapled evidence, caching and parallel requests change that ledger. A cached decision still has lookup/enforcement cost. Successful refresh at intervals no longer than twenty-four hours bounds cache staleness relative to available newly published status by roughly that interval, not the duration since an unreported compromise; outages, publication delays and policy must be handled explicitly. Do not cache mutable trust forever solely by asset hash. Vosoughi, Roy and Aral report that true Twitter stories took about six times as long as false stories to reach $1{,}500$ people in their studied cascades. That diffusion-time ratio does not estimate the probability that an unsigned asset is false or justify the chosen $0.2$ weight.[3]

36.4 Indirect prompt injection: the native RAG threat model

What it is

Direct injection arrives through an active user request; indirect injection arrives in lower-trust third-party material such as a page, email, PDF or retrieved passage. White-on-white or tiny text can be visible to an extractor while easy for a person to miss. Nikkei’s July 2025 investigation, subsequently described by The Register, identified seventeen preprints associated with fourteen institutions in eight countries; it was a targeted discovery report, not a representative corpus survey or a demonstrated success rate for every hidden prompt.[4][5] Role markers and message structure can encode instruction priority, and a trained model can respond differently to system, user and tool text. It is too strong to say all such text receives identical attention or that nothing marks the roles. The key limitation is that ordinary model context does not provide an independently enforced process privilege boundary. Wallace et al.’s learned instruction hierarchy improves scoped robustness without proving universal isolation.[6]

Why it exists

Injection concerns the source and authority of an instruction; jailbreaking concerns bypassing a safety restriction. They can overlap: a poisoned page may both redirect a task and seek an unsafe response. Good harmful-content refusal alone does not establish instruction-source authorization. Authentic or trusted-publisher content can also contain instructions that the application must treat as data.

Failure without it

Keyword and regular-expression filters can catch known strings but cannot be assumed to recognize all malicious natural-language intent; paraphrases and benign quoted imperatives create coverage/false-positive trade-offs. This is not a proof that all filters are useless or that SQL injection is solved by arbitrary escaping. Structured interfaces and parameterized SQL can enforce a code/data boundary for their stated grammar; an analogous boundary must be designed and validated for the LLM application, especially before privileged tool calls.

Cost, complexity, and decisions

For a mathematical exposure toy, let each of $k$ IID passage draws contain an injection with probability $p=0.0005=0.05\%$. Treat $p$ as the selected-draw rate under this hypothetical mechanism, not an empirically known global-corpus rate. If exposure means at least one injected passage, then:

At $k=10$, under the same IID model:

At $k=50$:

For small $kp$, $E(k)\approx kp$; the union bound $E(k)\le kp$ also holds without independence if each selected position has marginal probability at most $p$. Corpus prevalence alone does not establish those marginals. Under the IID model, requiring $E(k)\lt 0.001$ yields the strict inequality below because $\ln(1-p)\lt 0$:

Thus $k\le2$ in this toy: $E(2)=0.099975\%\lt 0.1\%$, while $E(3)\approx0.1499250125\%$. Do not round the strict threshold to two before deciding integer feasibility. A targeted nearest-neighbor query can retrieve only attack passages even when their global fraction is $0.05\%$; correlated draws and finite-population sampling require their own models. The hidden-preprint investigation gives no representative rate from which a “five times smaller” corpus prevalence or conservative risk floor can be derived. Exposure also differs from attack execution/success. Treat retrieved content as lower-trust data, keep roles/delimiters explicit and use hierarchy-aware models where evaluated. Scan the actual contexts supplied to generation, including reranked/expanded text, but do not call a passed classifier proof of safety. Keep tool privileges, data authorization, output handling and human approval for consequential actions outside solely model-generated decisions. Select retrieval depth for measured recall, latency and attack risk rather than claiming $k=50$ is safe after a scan. Output checks can still help prevent an action or response, though they cannot undo disclosure of restricted input already sent to a model.

36.5 Data poisoning the datastore

What it is

Datastore poisoning inserts or alters stored evidence, potentially affecting repeated queries until the records, derived chunks and caches are excluded. Indirect injection can be delivered through those persistent records; it is not necessarily ephemeral or disjoint from poisoning. Poisoning can instead manipulate factual answers without an explicit instruction. A local nearest-neighbor attacker targets selected queries rather than needing a large global corpus share, although dense systems can also incorporate authority signals and classical search can suffer local spam. PoisonedRAG injects five malicious texts per target question in its evaluated setting, not five total texts for an arbitrary workload. For example, its black-box PaLM 2/NQ result reports $97\%$ ASR with $2{,}681{,}468$ clean texts in the database. ASR is the fraction of target questions yielding the attacker-chosen answer under that protocol, not the fraction of arbitrary production requests compromised or a guarantee against every defended system.[7]

Why it exists

The selected evidence can concentrate malicious records far above their corpus fraction. A per-source ballot cap limits repeated records under one identity but does not establish independent ownership, observations or errors. Distinct domains can be coordinated Sybils; conversely, genuinely independent sources need not have independent errors. Generative-engine optimization includes legitimate discoverability work as well as possible manipulative practices, so distinguish intent, falsehood and unauthorized control instead of equating all GEO with poisoning.

Failure without it

If all five inserted records win a five-document window, their document-count share is $5/5=100\%$. With twenty selected records it is at most $5/20=25\%$ under the assumption that only those five are malicious; token/attention share can be very different. A reranker may promote or reject them, so dilution is not a security proof. PoisonedRAG also reports residual ASR at $k=50$ in a specific HotpotQA expansion experiment; more context alone did not eliminate that attack.[7] An illustrative classical campaign of $100$ pages versus five texts is $20{:}1$, or $\log_{10}20\approx1.301030$ orders of magnitude—not two orders. These stipulated counts are not matched attack-cost measurements, a universal link-farm floor, or a dollar-cost comparison. Creation/search costs, target coverage, ingestion, persistence and defenses all matter.

Cost, complexity, and decisions

RA-RAG is a possible source-level control, not a demonstrated defense against every poisoning/Sybil attack. Its accepted-source target $K=4$ out of $S=1{,}000$ can require more than four source attempts, and it transforms source agreement into vote weights rather than simply treating every rate as a probability. Under the original uncapped transform $v_s=S r_s-1$, a near-zero $r_s$ produces a negative vote—not a neutral unknown-source prior. Define unknown-source admission and calibration explicitly rather than silently reusing that transform for arbitrary cold-start trust.[8] Validate source control/independence, quarantine suspicious writes where appropriate, maintain provenance for chunks and invalidate serving caches on removal. Audit bursts of new sites repeating unusual claims without assuming every news cycle is an attack. Domain age or verified identity are signals, not truth proofs. The chapter’s five-blog/nine-day support scenario and eight-domain/two-week escalation are illustrative adversarial cases, not independently verified experiments or universal attacker costs.

36.6 Jailbreaking, red teaming, guardrails

What it is

Jailbreaking seeks content disallowed by a model’s safety policy, while an unauthorized discount is also an application instruction/transaction-authorization failure; these categories can overlap but are not identical. A chatbot’s one-dollar price utterance is not evidence that a sale executed or was legally binding. Pretraining and later training jointly affect capabilities and refusal behavior. Competing goals and out-of-distribution role-play are useful explanations to test, not an exhaustive mechanism theorem or proof that every model already contains every requested harmful capability.

Why it exists

Use separate scans for active requests and retrieved third-party content when those controls help, plus enforcement on tools, data and output consumers. A system can fail an authorization boundary while maintaining ordinary harmful-content refusal. Layered defenses need not have independent errors, so do not multiply their individual failure rates without a justified joint model. Red-team examples improve the test set but cannot certify the complement of all unseen attacks.

Failure without it

A keyword/embedding bank can become stale; its useful lifetime is workload-dependent rather than necessarily a fixed number of months. Adaptive attacks may evade similarity, attack the scorer, span multiple turns or exploit false positives. No finite clean test proves zero attack risk. Do not automatically admit every flagged production query into the trusted bank: that creates a write/poisoning path. Review examples, permissions, duplicates, labels and retention, and evaluate on a separate held-out/adaptive suite.

Cost, complexity, and decisions

Consider a hypothetical guardrail that embeds a bank of $J=50{,}000$ attack patterns and compares each request with it. A match may choose a hardened prompt or an explicit review/block policy; temperature zero is not a safety guarantee or a promise of perfectly deterministic serving. The chapter’s unattributed $71\%\to2\%$ ASR pair is used here only as a stipulated scenario, not as a verified published benchmark or transferable production result. Let $Q=50{,}000$ requests per day and assume the request-level attack fraction is $a=0.001$. If the conditional per-attack success rates really are $0.71$ without and $0.02$ with the entire deployed defense—including missed detections—then expected successful attack requests are:

Under the same hypothetical workload and outcome definition:

The modeled difference is $34.5$ successful attack requests per day, a relative reduction $34.5/35.5\approx97.183099\%$, not a guarantee of exactly one incident each day. Distinct records, users, harms and multistep attack sessions need separate denominators. A $2\%$ success rate only on detected/matched prompts would not justify the full-defense calculation; include misses and the actual attack mix. Treat the proposed $10$–$20\,\mathrm{ms}$ embedding time plus a low-single-digit-millisecond ANN lookup as service assumptions to benchmark, not near-zero guaranteed latency for $J$ vectors. Include queueing, false-positive paths, context scans and any review. Route by severity and validated thresholds; hardening or blocking can both cost benign utility. Re-test affected model/prompt/retriever/scorer changes and stratify genuinely unrelated releases rather than using an unvalidated daily-versus-weekly schedule as a security rule. Admit new bank entries only after controlled review, and track attacks, misses, benign refusals, latency and adaptive residual risk.

36.7 Adversarial training: discrete, continuous, and hybrid

What it is

A discrete adversarial input $x_{\mathrm{adv}}$ is a realizable token sequence; training can pair it with an appropriate refusal target $y_{\mathrm{refuse}}$ while preserving benign utility. Token identity itself cannot be changed by an infinitesimal gradient step, but gradients through a relaxed one-hot representation can shortlist replacements. In the stipulated GCG-style search configuration, suffix length $L=20$, shortlist size $256$, candidate batch $B=512$ and $T=500$ iterations give $TB=256{,}000$ candidate-sequence forward evaluations if every iteration evaluates the full batch. This does not include the iteration’s gradient calculation, filtering, early stopping or batched execution details; it is not necessarily $256{,}000$ sequential accelerator launches. For an illustrative continuous objective, let $e=\mathop{\mathrm{Embed}}\nolimits(x)\in\mathbb R^{L\times d}$ and choose a global Frobenius norm budget over declared editable positions. With $\epsilon\ge0$, one refusal-loss adversary is:

An outer update minimizes a declared loss on the adversarial input, typically with benign utility terms. Approximate projected gradient ascent need not find the global maximizer. For a global ball, project a proposed $z$ as $z\min(1,\epsilon/\|z\|_F)$ when $z\ne0$, and leave zero unchanged. Ten or twenty inner gradient steps are example budgets, not universal optima; each typically involves forward/backward work and can be minibatched. Per-token normalized perturbations, global budgets, harmful-target objectives and preference losses are different conventions. Do not claim this pedagogical loss is literally every implementation of C-AdvUL, C-AdvIPO, CAT or MixAT.

Why it exists

An embedding ball includes unrealizable perturbations and need not contain distant textual attacks. Its geometry depends on embedding scale, norm, editable positions and sequence length; larger radius is not guaranteed to improve robustness or harm utility monotonically. Finite adversarial training does not certify every point in the ball. MixAT’s central construction applies continuous attacks around sampled discrete adversarial inputs, including persuasive paraphrases, rather than merely adding a few discrete-only anchor losses to otherwise clean-centered continuous training. Its default PAP mixture does not require an online GCG search at every anchor.[9]

Failure without it

Discrete search can be costly, but whether it is infeasible depends on the search, caching, batching, behavior set and budget. Continuous-only robustness can fail on real attacks outside its trained perturbation space, while a hybrid can still miss untrained attacks and lose benign utility. For $n\ge1$ common test behaviors and a finite nonempty attack set, let $S_{b,j}\in\{0,1\}$ record success of method $j$ on behavior $b$ under its declared attempt budget. Define $\mathrm{ASR}_j=n^{-1}\sum_b S_{b,j}$. Then $\mathrm{ALO\!\!\!-ASR}=n^{-1}\sum_{b=1}^{n}\mathbf1[\exists j:S_{b,j}=1]$. It is the fraction of behaviors broken by at least one tested attack, not the fraction of attack methods that break something. It satisfies $\max_j\mathrm{ASR}_j\le\mathrm{ALO\!\!\!-ASR}\le\min(1,\sum_j\mathrm{ASR}_j)$ without independence. It remains a suite- and budget-dependent union, not universal worst-case security.[9]

Cost, complexity, and decisions

Assuming an amortized $5\,\mathrm{ms}$ per candidate forward evaluation and no early stop, the GCG candidate component is $256{,}000(0.005)=1{,}280\,\mathrm s\approx21.333333\,\mathrm{min}$ per suffix. Five hundred separately searched framings cost $640{,}000\,\mathrm s\approx10{,}666.666667\,\mathrm{min}=177.777778\,\mathrm h\approx7.407407\,\mathrm{days}$ of serial-equivalent candidate work, excluding gradient/search overhead, training and refresh decisions. A universal reused suffix would have a different amortization. Ten continuous inner gradient steps give the numerical count ratio $256{,}000/10=25{,}600$, but a candidate forward evaluation is not a forward/backward step: this is not a measured cost or wall-speed ratio. Use a versioned primary result for comparison. MixAT v1 Table 1 reports Zephyr-7B ALO-ASR $67.5\%$ for the retrained CAT baseline and $15.0\%$ for MixAT on the first forty non-copyright HarmBench behaviors; its MixAT GCG ASR is $12.5\%$ and HumanJailbreaks ASR $5.0\%$. The paper’s prose also says $12.5\%$ ALO, but the cited table places that number in the GCG column; this chapter uses the explicit table’s $15\%$ union. Llama3-8B MixAT ALO is $25\%$ in that table, so “under twenty percent” is not universal across architectures. These reported experiments are not replicated here.[9] Table 2 reports Zephyr MixAT training at four hours on two A100s versus six hours forty minutes for CAT, but with different step counts ($300$ versus $760$) and recipes. That is a reported run-cost comparison, not isolated equal-step efficiency. The default discrete seeds are PAP attacks; adding GCG is a separate variant with greater search cost. Choose/refine the perturbation scale and anchor mixture on held-out real attacks and benign utility, refresh when measured validity changes, and report per-attack ASR, ALO-ASR, sampling uncertainty, judge/budget settings and full training/search cost.[9]

36.8 Privacy: what the datastore protects and what it does not

What it is

A live datastore row can be selected and blocked from future serving if all read paths enforce that state; replicas, backups, derived indices, caches and prior disclosures need separate treatment. The recalled approximately $\$1.338$ million and $22.30$-day full-retraining estimate is an idealized chapter-five compute baseline, not a universal deletion price or comparison with an identical output unit in extraction. Revocability and present confidentiality answer different questions. Define $\mathrm{Allowed}(u,c)\in\{0,1\}$ from authenticated requester $u$, chunk $c$ and current policy. The intended retrieval domain is $\{c\in D:\mathrm{Allowed}(u,c)=1\}$. Similarity $\mathrm{sim}(q,c)$ alone does not enforce that predicate.

Why it exists

First, relevance-only retrieval can select records forbidden to this requester. Second, sending selected context to a provider or tool not authorized to process it is a disclosure regardless of whether it trains on the data; an approved processor can instead be within the declared security arrangement. Third, embeddings can reveal sensitive text. Morris et al. report exact recovery of $92\%$ of thirty-two-token inputs under their evaluated embedding-inversion setup. That is evidence of a serious attack surface, not guaranteed reconstruction of every embedding/document. Protect vector access conservatively and evaluate the actual encoder, attack access and input lengths.[10] OWASP’s 2025 vector-and-embedding category identifies unauthorized access and data leakage from inadequate controls. None of these paths requires a weight update, though the generator’s weights can affect what is emitted. A general report of confidential text sent to a hosted service does not by itself prove later training use or disclosure outside the service’s actual processing agreement.[11]

Failure without it

Not fine-tuning on a record avoids that training path; it does not prove no prior training exposure or close retrieval, provider, cache, logging or inversion paths. Output redaction cannot undo restricted context already sent to an unauthorized model, and paraphrases can evade literal filters. Distinguish output filtering from a trusted authorization check applied after candidate ranking but before forbidden chunks are fetched or exposed to the model: that latter check can preserve confidentiality if correctly enforced, although a fixed top-$k$ may underfill. Integrating permission-aware selection earlier can avoid wasted slots; it is not the only logically possible zero-forbidden-context enforcement point. Identity, metadata correctness, stale permissions, trusted search internals and side channels still matter.

Cost, complexity, and decisions

Stipulate a corpus of $N=200{,}000$ chunks with $R=20{,}000$ marked restricted. A candidate pool of $M=50$ has an expected five restricted items only under an appropriate base-rate sampling assumption; deterministic semantic ranking need not preserve that fraction. For a separate conditional toy, fix exactly $K=5$ restricted candidates in the pool and choose $k=8$ uniformly without replacement. Then $X$, the restricted count selected, is hypergeometric—not an actual deterministic top-eight risk model:

The one-decimal risk is $59.9\%$ in that conditional random-draw toy. It is not a conservative floor for actual salary queries without a justified selection model. At $k=1$ the toy risk is $5/50=10\%$; at $k=50$ it is one. If eight items were sampled uniformly directly from the entire fixed corpus instead, the risk would be $1-\binom{180{,}000}{8}/\binom{200{,}000}{8}\approx56.953949\%$, a different model. Conditioning on exactly five among fifty is not equivalent to substituting an expected count into every selection process. If authenticated identity, ACL metadata and enforcement are correct, excluding all forbidden chunks before exposure makes the restricted-context selection probability zero by construction. That conditional invariant is not proof of zero system-wide privacy breaches. It does not cover side channels, stale labels, policy bugs, other tools, preexisting weights or downstream retention. A trusted post-ranking check before disclosure can also enforce it; pre-filtered ANN additionally needs workload-specific recall/latency testing rather than a guaranteed single-digit-millisecond surcharge. Carry and synchronize source permissions onto chunks, metadata, caches and derived stores; fail safely on unknown authorization. Minimize context logging and protect vectors like sensitive source data. Define approved provider/process boundaries and retention terms explicitly. Fuzz the same questions across authorized/unauthorized scopes, revoked users, changed groups, stale caches, errors and concurrent updates; tests support the enforcement claim without proving all possible executions. A three-product-line shared index is a deployment scenario, not an exception to the requester-specific rule.

Diagrams

Figure 36.1

Figure $36.1$: Attribution and provenance are different evaluation axes, not a claim of statistical independence. High support with excluded provenance is an important failure case; high scores on both axes still do not establish factual truth. Missing/unknown provenance is not automatically a proven fake.

Figure 36.2

Diagram description and source

flowchart TD
  A["Asset portions covered by format-specific hard binding"] --> H["Recompute and check content binding"]
  M["Manifest: assertions, claim and claim signature"] --> S["Verify correctly serialized signed claim"]
  M --> I["Ingredient references: available provenance graph"]
  I --> P["Prior manifests or multiple ingredients; capture may be absent"]
  S --> T["Certificate path, relevant time/status evidence and trust policy"]
  H --> R["Report binding, signature and policy outcomes separately"]
  T --> R
  P --> R

Figure $36.2$: A simplified C2PA validation graph. Assertion count is not signature count; a history need not be a complete linear capture chain. A hard-binding mismatch can coexist with a valid claim signature, and an untrusted signer’s signature can verify. Authentication of assertions does not prove their factual truth.[2]

Figure 36.3

Diagram description and source

flowchart TD
  D["Asset and discovered provenance evidence"] --> M{"Usable manifest available?"}
  M -- "no" --> U["Unknown or unavailable: declared coverage policy, not proven forgery"]
  M -- "yes" --> C["Check bindings, references and signature separately"]
  C --> V{"Cryptographic checks pass?"}
  V -- "no" --> F["Binding mismatch or invalid signature: report precise cause"]
  V -- "yes" --> T["Evaluate trust scope, relevant times and status policy"]
  T --> P["Accepted, policy-excluded or unresolved"]
  P --> A["Anomaly signal may request review; not proof of compromise"]
  U --> W["Rank, label or exclude under explicit policy"]
  F --> W
  A --> W

Figure $36.3$: Missing, invalid, untrusted, compromised and anomalous credentials are distinct states. A known compromised key can sign a cryptographically valid claim. An anomaly alone does not justify calling the item forged or blocking the signer. Ranking weights are declared policy values, not truth probabilities.

Figure 36.4

Diagram description and source

flowchart TD
  S["Higher-priority application instructions"] --> M["Model with explicit roles and learned hierarchy"]
  Q["Authorized user task"] --> M
  A["Lower-trust retrieved evidence A"] --> M
  B["Lower-trust evidence B contains unauthorized instruction"] --> M
  M --> O["Proposed answer or tool action"]
  O --> E["Independent authorization, least privilege and output/action checks"]
  E --> R["Permitted result or refusal/review"]

Figure $36.4$: Message roles can mark priority and alter learned behavior; attention is not identical for every instruction. Their presence does not itself enforce application privilege or confidentiality. External authorization and tool controls remain necessary even with a hierarchy-trained model.[6]

Figure 36.5

Diagram description and source

flowchart TD
  D["Datastore with mostly honest text and a few query-targeted malicious records"] --> R["Rank candidates for the targeted query"]
  R --> K["Selected malicious slot share can greatly exceed corpus prevalence"]
  K --> G["Generator sees selected text; slot share is not attention share"]
  I["Poison can be false evidence, indirect instructions or both"] --> D
  K --> C["Validate provenance, source dependence, instruction handling and outcomes"]

Figure $36.5$: Targeted retrieval does not preserve global attack prevalence. Five malicious texts are a per-target-question budget in PoisonedRAG’s cited experiment. Their count share alone does not determine token share, attention or ASR, and removal must cover derived serving state.[7]

Figure 36.6

Diagram description and source

flowchart TD
  Q["Incoming query"] --> G["Guardrail scorer and reviewed pattern bank"]
  G --> P["Validated normal, hardened, block or review policy"]
  C["Retrieved content scan and permission enforcement"] --> M["Model"]
  P --> M
  M --> O["Output and privileged-action controls"]
  R["Controlled red team and reviewed production near-misses"] --> B["Curated bank updates"]
  B --> G
  R --> A["Adversarial training plus benign-utility checks"]
  A --> M
  O --> V["Monitor misses, false positives, harm and latency"]

Figure $36.6$: Inline scoring has a measured workload-dependent cost, not guaranteed near-zero latency. Review new patterns before admission, evaluate the full defended path and keep tool/data authorization separate. Defense errors can correlate; layering does not justify multiplying isolated ASRs.

Figure 36.7

Figure $36.7$: Conceptual union of continuous neighborhoods centered on discrete inputs. Here $\mathcal R(x)$ contains the clean input and sampled attack variants embedded in a common padded/editable-position layout for the illustration; actual norm/masking conventions must match the implementation. MixAT uses continuous perturbations around sampled PAP attack seeds as well as its original-input mixture, not a mandatory fresh GCG search at every isolated anchor. A finite training run does not certify the whole union.[9]

Figure 36.8

Diagram description and source

flowchart TD
  Q["Authenticated requester and query"] --> A["Current chunk authorization policy"]
  D["Shared datastore and vector index"] --> R["Trusted permission-aware search or pre-exposure authorization"]
  A --> R
  R --> C["Only allowed context reaches authorized consumers"]
  C --> G["Approved model/provider processing boundary"]
  G --> O["Response, tool and retention controls"]
  D --> V["Protect vector reads against inversion/data leakage"]
  A --> S["Synchronize permission changes, caches and derived stores"]
  S --> R

Figure $36.8$: Deletion and authorization are separate controls. A correct permission filter can ensure no forbidden chunk enters model context under its assumptions; it does not prove zero leaks through every system path or recall past disclosures. A trusted pre-exposure post-ranking check can also enforce confidentiality, with possible underfill.

Whiteboard pack

What to draw

1. Draw separate attribution, provenance, factual-credibility and authorization checks; do not label them statistically independent.
2. Show an asset hard binding and a signed claim with assertion references. Draw an ingredient graph, with unknown earlier history explicitly possible.
3. Split outcomes into missing/unavailable, binding mismatch, invalid signature, untrusted/status-excluded, anomalous and accepted.
4. Write the partial verification ledger: $6+0.9=6.9\,\mathrm{ms}$; keep historical hashes and status checks outside that component budget.
5. Draw higher-priority instructions and lower-trust retrieved text feeding a model with role markers. Put privilege enforcement outside the model.
6. Mark the difference between corpus prevalence and a targeted top-$k$ window; five per-question poison texts need not be a five-text workload budget.
7. Write $E(k)=1-(1-p)^k$ only beside an IID sampling label; mark exposure different from success.
8. Draw guardrail, reviewed red-team feedback and adversarial training with correlated possible failures.
9. Draw continuous neighborhoods around clean and sampled discrete attack inputs. Separate candidate-forward counts from gradient steps and wall cost.
10. End with the authorized candidate domain and a pre-exposure enforcement boundary. Label the $59.850667\%$ hypergeometric figure a conditional randomized toy, not a top-eight guarantee.

Spoken script

RAG trust has distinct jobs. Attribution checks textual support; provenance validates specified signed claims and content bindings; neither proves factual truth. Retrieved instructions can come from the wrong authority even when the file is authentic, so role-aware models need independent data and tool permissions. Poisoned evidence can concentrate in a query’s selected window without a large corpus share. Guardrails, red teams and adversarial training address different weaknesses, but finite tests and correlated failures leave residual risk. Finally, deletion is not confidentiality: verify current permission before any restricted chunk reaches an unauthorized consumer, and handle caches, logs, vectors and providers as part of that boundary.

Interview traps

1. A quote exactly supports the answer. Does that establish provenance?

No. Exact quotation can establish textual support without authenticating a source. In the stipulated toy, attribution scores $0.97$ and $0.94$ do not diagnose provenance. Validate the asset/credential evidence and policy separately, and remember that a valid signed document can still be false or fail to support this claim.

2. Five independently named sites agree on a new claim. Why not trust the majority?

Different names or domains do not prove independent ownership, observations or errors. A ballot cap restricts one identity, not Sybils. Treat new-source admission as an explicit policy, seek independently validated track records and audit coordinated bursts without turning ordinary news repetition into automatic guilt. Unknown reliability is not the same as a near-zero RA-RAG agreement rate.

3. Is indirect prompt injection just another jailbreak?

They are different properties and can overlap. Indirect injection delivers an unauthorized instruction through third-party content; jailbreaking seeks a safety-policy violation. Test incoming requests and retrieved content, but also enforce least privilege and authorization outside model prose. Good refusal on one suite proves neither complete instruction isolation nor confidentiality.

4. Why use guardrails, red teams, and hybrid adversarial training together?

Guardrails score known or generalized patterns, red teams reveal misses, and adversarial training changes behavior on a chosen attack space while needing benign-utility checks. The $25{,}600$ figure is a ratio of candidate-forward evaluations to continuous gradient steps, not a measured cost ratio. MixAT centers continuous attacks on sampled discrete inputs; report its per-attack and ALO metrics with the actual model, suite and budget.

5. When would you not apply every strongest control?

Choose additional provenance depth, scoring, review and retraining by the stated risk and measured cost; first-party ownership alone is not an integrity guarantee. Never let restricted content reach an unauthorized consumer. A trusted permission check after ranking but before fetch/disclosure can enforce that rule, whereas filtering only the generated output cannot undo input disclosure. Permission-aware retrieval can also improve candidate utilization; benchmark its actual overhead.

Key numbers

00-START-HERE · 01-READING-ROUTES · 35_source_credibility · 37_latency_cost_and_systems