Chapter 13: Chunking and Granularity

00-START-HERE · 01-READING-ROUTES · 12_text_representation · 14_beyond_plain_text_tables_layout_documents This chapter explains how to choose retrieval units, route across granularities, preserve enough context, and measure whether a chunker actually improved retrieval.

TL;DR

• Chunking is an index-time decision. A single-chunk retrieval path may lose a split fact; neighboring-chunk retrieval or parent expansion can repair it if the underlying text remains available.
• Start from the smallest unit that must survive intact. Set overlap from that unit’s length, not from a fixed percentage.
• Use boundaries already present in the document before paying a model to infer them. Meaning-based and model-generated splitting can cost $3.5\times$ and about $45\times$ as much as fixed splitting.
• Query type determines useful granularity. Do not assume raw sentence, paragraph and section scores are calibrated for one ranking; validate or calibrate them before fusion.
• Fine retrieval helps when pooling hides one exact fact. Index a self-contained proposition, then return its parent passage so the generator still sees qualifiers.
• Evaluate retrieved extent with token recall, precision, and intersection over union ($\mathop{\mathrm{IoU}}\nolimits$). Recall alone rewards oversized chunks.
• Semantic chunking needs a measured benefit over structural cuts and overlap. In the Gaussian examples, rounded percentile-like thresholds give approximately $100\%$ versus $31\%$ boundary recall; this is a model comparison, not a universal transfer bound.

The story

Imagine a library that receives long policy binders. A cutting clerk prepares searchable cards before patrons arrive. If a fact is split across cards, a desk that retrieves only one card may miss it; a desk that also fetches the neighbor or parent can reconstruct it. Otherwise a new cutting policy requires reindexing affected content. The cheapest clerk counts tokens and cuts on schedule. If a clause crosses the cut, its halves land on different cards. Overlap photocopies the edge of one card onto the next. In the discrete window model, an overlap of $o$ tokens protects every span of at most $o+1$ tokens, provided the span fits within a chunk. A more careful clerk follows paragraph breaks. A better clerk reads the binder’s headings, table rows, and clause numbers. Those author-provided marks usually reveal the intended units for almost no extra work. A meaning-aware clerk listens for topic changes. A model-driven clerk reads the whole binder and proposes cuts. Those last two clerks cost more, and they can rediscover boundaries the author already supplied. Some binders contain useful policy text wrapped in repetitive legal boilerplate. A local clerk compares each sentence with its neighbor and cuts when the subject changes. A pseudo-instruction clerk first writes a one-card summary of the binder. The clerk then compares every sentence with that summary and separates runs when usefulness changes. The summary is a proxy for future patron questions. The library also keeps cards at sentence, paragraph, and section sizes. A dosage question wants the sentence card. A broad recommendation question wants the section card. Raw catalog scores from these shelves are not comparable because long, coherent cards can inherit a high background score. The library therefore uses a router, meaning a clerk who reads the question and weights each shelf before ranking. The finest cards hold propositions, meaning one self-contained fact each. A sentence such as “the fee rises to $\$780$ that year” is too dependent on its neighbors. The proposition card rewrites it with the subject, date, and value. The search desk matches that precise card, then follows its parent pointer back to the original passage. The parent passage matters because the answer clerk is a large language model (LLM), and the LLM needs conditions and qualifiers. The proposition is the match unit. The passage is the serve unit. If the library serves only the proposition, a true fragment can still support a wrong answer. The head librarian audits each cutting policy against highlighted gold excerpts. Token recall asks how much highlighted text came back. Token precision asks how much returned text was highlighted. $\mathop{\mathrm{IoU}}\nolimits$ charges for both missing text and padding. Returning the correct whole binder gives perfect extent recall but nearly useless precision; the retriever must still find that binder. Finally, the head librarian tests the meaning-aware clerk on two shelves. One shelf joins unrelated articles. Topic changes are obvious, so the clerk finds almost every seam. The other shelf contains single coherent policies. Adjacent sections share vocabulary and style, so real seams look like ordinary sentence gaps. The same detector now cuts mostly in the wrong places. The library’s rule is practical. Use declared structure first. Buy overlap when it protects a measured answer span. Add fine cards only when pooling truly dilutes exact facts. Add a router only when queries genuinely need different scales. Make every expensive clerk win on real binders before funding a full recataloging.

Decoder table

Core mechanics

Mathematical conventions

Token counts and span lengths are integers. For fixed overlapping windows use $1\le L$, $0\le o\lt L$, retain the final partial chunk, and stop as soon as the document is covered. Counts and billed unpadded tokens are computed per document, not by pooling unrelated documents into one stream. Long-document overlap multipliers are approximations; finite vector counts and padded compute can differ. A split span can still be recovered by retrieving neighboring chunks or expanding to a parent: failure of a single chunk is not a proof that all RAG systems must fail. Vector-pooling and Gaussian examples are explicitly idealized, not laws of learned encoders. Natural logarithms use nats; storage is decimal. HNSW budgets include vector payload and base-layer neighbor slots only. FLOP estimates use stated sustained throughput and omit attention-length effects, bandwidth, padding unless specified, scheduling, graph work, and I/O. Extent metrics use unique source positions keyed by document, not repeated prompt tokens. Keep full precision until reporting; $\approx$ denotes rounding or a model approximation.

13.1 Five levels of splitting

The index-time retrieval contract

• What: Splitting chooses searchable units before retrieval runs; the text itself need not be compressed or discarded.
• Why: A single-unit match should contain the needed evidence, or the serving path must explicitly recover adjacent or parent context.
• Failure without it: A $900$-token clause cut at $512$ can separate obligation from liability cap. A reranker cannot invent a missing candidate, but retrieving both halves can repair the split.
• Cost and complexity: Boundary changes invalidate affected chunk IDs and vectors. Version the chunker and its parent mapping; an entire index rebuild is not always necessary. For a nonempty document with $T$ tokens, chunk length $L$ and overlap $o$, let stride $S=L-o$. Chunk starts are $0,S,2S,\ldots$ until a chunk first covers the document end. The resulting count is

Set $n=0$ for an empty document. With an unpadded final chunk and no extra windows after coverage, the exact embedded-token count is $T+(n-1)o$ for $T\gt 0$. The long-document approximation is $T\,L/(L-o)$; padding every call to $L$ instead processes $nL$ slots. If span starts are uniform modulo the stride, ignoring document-edge effects, the probability that no one window contains an $s$-token span is $\min(1,\max(0,(s-o-1)/(L-o)))$ for $1\le s\le L$; it is $1$ for $s\gt L$. At $o=0$ this becomes $(s-1)/L$ for a fitting span. At $s=40$, $L=512$, and $o=0$, modeled severance is $39/512=0.076171875\approx7.62\%$. Any span with $s\le\min(L,o+1)$ is contained in at least one window under this discrete coverage convention. Thus $o\ge39$ is sufficient for every $40$-token span; $o\ge40$ is a conservative alternative. The long-document multiplier $L/(L-o)$ approximates token and vector inflation. Exact token count is $T+(n-1)o$, exact vector count is $n$, and graph allocation follows that integer count. These need not share exactly one multiplier on short documents. At $L=512$, $o=64$, the asymptotic multiplier is $512/448\approx1.142857$, or about $14.286\%$ extra. For the finite contract example below, actual token inflation is $4{,}512/4{,}000=1.128$.

The five splitting levels

With one neighboring sentence on each side, interior sentences appear in three detection windows. Edge sentences appear fewer times. Thus detection is approximately $3T$ for long documents, and final disjoint chunk embeddings add $T$; exact cost depends on edges, padding and the implementation. A strict cut above the empirical $95$th percentile selects approximately the largest $5\%$ of distances when values are distinct. Ties, small samples and the quantile convention affect the count; identical distances need not produce any strict-above-threshold cut. A fixed top-fraction rule can nevertheless manufacture boundaries on noise. Level five earns its cost mainly when markup is absent. The source names OCR scans, ASR transcripts, and raw plain text. Doubling $L$ halves the no-overlap severance probability for a fixed fitting span under uniform phase. It also roughly doubles retrieved text per full chunk and the linear prefill proxy. An individual final token vector has coefficient $1/L$ in a mean, but that is not its end-to-end influence in a contextual encoder. Cross-length score bias depends on the geometry discussed below.

Worked corpus

The corpus has $100{,}000$ documents at $4{,}000$ tokens each. Total $T$ is $4\times10^{8}$ tokens. Embedding costs $\$0.02$ per million tokens. The generation model costs $\$1.00$ per million input tokens.

At $5\%$ weekly turnover, proportional recomputation costs $0.05(\$408)=\$20.40$ weekly under the same input-plus-embedding model, excluding generation output charges. Dense Passage Retrieval (DPR) used $21{,}015{,}324$ disjoint 100-word English Wikipedia passages. At about $1.33$ tokens per word, $L$ is about 133. A 40-token span has modeled severance $\frac{39}{133}=29\%$. A 5-token short answer has modeled severance $\frac{4}{133}=3.0\%$. The crude splitter fit that task because the required unit was short.

Practical decisions

• Default to level three when markup exists. In the worked corpus it is structurally faithful and costs $\$8.00$ instead of $\$9.024$ in the exact finite-window comparison.
• Set $o$ from measured required-span length. Protecting every $40$-token span requires $o\ge39$ under this discrete model.
• Increase $o$ only after sampled failures show longer spans. The source proposes 100 sampled failures.
• Index subchunks and return the parent when a structural unit exceeds the encoder’s useful length.
• Cap parent expansion when top-$k$ parents overrun the context budget.
• Buy level five for high-traffic documents first. The example applies it to $5\%$ of documents for $\$20.40$ instead of a $\$408$ full pass.
• For fixed $N$, $d$ and index settings, chunk token length does not directly change the arithmetic of one vector comparison. Changing the chunker can still change vector count, geometry, recall and ANN work, as well as generator prefill and its tail.

13.2 Chunking by intent: pseudo-instructions

The conditioning problem

• What: A document is an ordered sentence sequence $D=(s_1,\ldots,s_n)$. A chunker chooses boundary positions $B$ from 1 through $n-1$.
• Why: The ideal $B$ would keep every sentence needed by query $q$ in one run and exclude everything else.
• Failure without it: The ideal boundary depends on $q$, but the index must commit boundaries before $q$ exists.
• Cost and complexity: Every practical chunker must choose a proxy for the unknown query. Fixed splitting uses length and a fixed phase, but no semantic query-intent estimate. Breakpoint chunking compares each sentence $s_i$ with neighbor $s_{i+1}$. It cuts on local subject change. Pseudo-instruction chunking generates a document summary $g$ and compares every sentence with that global anchor. For each sentence,

The threshold is the document mean

The binary label is $z_i=1$ when $r_i\ge r_{\mathrm{mean}}$ and 0 otherwise. The boundary set is every i where $z_i$ differs from $z_{i+1}$. This labels runs of above-mean and below-mean similarity to a summary proxy, not verified usefulness for an unknown future query.

Why the summary anchor matters

• What: The generated summary estimates what future queries are likely to seek.
• Why: It is salience-weighted rather than frequency-weighted.
• Failure without it: A local breakpoint can fuse adjacent boilerplate because the boilerplate is internally similar. A centroid can point at boilerplate when boilerplate is most of the document.
• Cost and complexity: The summary requires an LLM pass. Real attributed query logs are better when available. Let $S$ be the sum of unit sentence vectors. When $S\ne0$, define the normalized centroid $S/\lVert S\rVert$. The sum over sentences of $\cos\!\bigl(e(s_i),\mathop{\mathrm{centroid}}\nolimits\bigr)$ equals $\lVert S\rVert$. The mean centroid score is $\frac{\lVert S\rVert}{n}$. It measures document homogeneity, not document intent. The reported ablations replace the true summary with another document’s summary, a random document sentence, or the average embedding. All three degrade retrieval. The claim limit is that the published evaluation is general-domain. A specialist query distribution may emphasize an attribute no summarizer includes. The mean threshold always creates at least one cut when scores are not identical. Since $\min_i r_i \lt r_{\mathrm{mean}}\le \max_i r_i$, a single-topic page can split on noise. Guard with $\max_i r_i-\min_i r_i$. Emit one chunk when the spread is below a threshold calibrated on a held-out set.

Worked policy example

The corpus has $200{,}000$ policy documents. Each has 40 sentences at 22 tokens each, or 880 tokens. Twelve sentences contain policy. Twenty-eight contain preamble, revision history, and footer material. The toy geometry takes $u,v,w,\{\epsilon_i\},\{\delta_j\}$ to be mutually orthonormal directions. Each policy sentence has vector $0.8u+0.6\epsilon_i$; each boilerplate sentence has $0.8v+0.6\delta_j$. The query is $q=u$. A unit summary direction $g=0.9u+0.2v+\sqrt{0.15}\,w$ gives policy similarity $0.72$ and boilerplate similarity $0.16$. Assume all $12$ policy sentences form one contiguous run. For the mixed-chunk comparison, take a $20$-sentence, $440$-token unit under a $512$-token cap; it is not an exact blind $512$-token window. For the general version, write each policy vector as $\alpha u+\sqrt{1-\alpha^2}\epsilon_i$ and each boilerplate vector as $\beta v+\sqrt{1-\beta^2}\delta_j$, with $0\le\alpha,\beta\le1$. Here $\alpha=\beta=0.8$. A normalized sum of $a$ policy and $b$ boilerplate vectors, for $a+b\gt 0$, has

The mixed target’s modeled expected rank is about $417$, beyond a rerank depth of $100$. This is not a measured rank. Gaussian cosine tails are local approximations rather than valid unbounded cosine distributions, and recomputing a threshold from noisy scores changes the flip calculation.

Cost and operating rules

The full corpus contains $1.76\times10^{8}$ tokens. A $110$-million-parameter encoder’s sentence pass uses $2NT=3.872\times10^{16}\,\mathrm{FLOP}$, or $113.882\,\mathrm s$ at $3.4\times10^{14}\,\mathrm{FLOP/s}$. This one-pass baseline excludes summary-vector encoding and final chunk re-encoding. An $8$-billion-parameter summarizer uses $2.816\times10^{18}$ prefill FLOPs and $2.56\times10^{17}$ decode FLOPs for $80$ output tokens per document. Total work is $3.072\times10^{18}\,\mathrm{FLOP}$, or $2.50980$ GPU-hours: $79.3388$ times the one sentence-embedding pass. Summary-anchor embeddings add $3.52\times10^{15}$ FLOPs if all $80$ generated tokens are encoded; final chunk embeddings add another pass over their actual token count. These are compute proxies, not latency measurements. Prefill is $2.816/3.072=11/12\approx91.667\%$ of this summarization work.

• Cache the pseudo-instruction on a content hash.
• At $2\%$ turnover, summarization uses about $0.02\times79.3388\approx1.587$ times a full-corpus sentence-embedding pass. Relative to a changed-documents-only sentence pass it remains about $79.34$ times; name the baseline.
• Use real queries with document attribution where they exist. Use summaries for cold start.
• Avoid PIC for short single-topic documents with no intent structure.
• Cap each intent run after the relevance cut. Split an overlong run at the nearest paragraph break.
• Score the chunker with $\mathop{\mathrm{IoU}}\nolimits$ against answer spans. $\mathop{\mathrm{recall}}\nolimits @k$ alone is confounded by encoder and ANN settings.

13.3 Mixture of Granularity: routing over chunk sizes

Why one tuned size fails

• What: The source sweeps 128, 256, 512, and 1,024-token chunks with $20\%$ overlap on 400 questions.
• Why: Factoid queries need fine evidence. Thematic queries need broad evidence.
• Failure without it: The aggregate winner at 512 serves neither query population optimally.
• Cost and complexity: Multi-level indexes add vectors, searches, a router, and a context-budget decision. Index nested sentence, paragraph, and section units. Let $a$ be query similarity to the one answering sentence. Let $b$ be similarity to each non-answering neighbor. Let $\rho$ be mean pairwise sentence similarity. For an $n$-sentence chunk:

For $\rho\gt 0$ and a feasible family of unit-vector geometries, $s(n)\to b/\sqrt\rho$ as $n\to\infty$. Feasibility constrains the parameters; in the limit $b^2\le\rho$. The expression is not a law for arbitrary contextual embeddings or for $\rho=0$. For $n\gt 1$, a coarse chunk beats the answer sentence when $b\gt a[\sqrt{n(1+(n-1)\rho)}-1]/(n-1)$. The simpler $b\gt a\sqrt\rho$ compares the limiting score with $a$, not every finite chunk. With $a=0.62$, $b=0.50$, and $\rho=0.5$:

A global top-2 list returns the correct section and the wrong section. It drops the exact answer sentence. For a thematic query with $a=b=0.45$, sentence, paragraph, and section scores are $0.450$, $0.569$, and $0.617$. The same coarse bias now makes the correct choice, which helps the bug survive review.

Router and fallback choices

• What: MoG feeds the query embedding to a small network that emits one weight $w_g(q)$ per granularity.
• Why: It rescales each level before candidates compete.
• Failure without it: Raw similarities encode length and relevance together.
• Cost and complexity: The router needs downstream answer-quality supervision and creates a query-conditioned failure surface. Each candidate receives weighted score

Top-$k$ selection is not differentiable. The source therefore describes a soft training target over levels, not a gradient through retrieval. Under the displayed multiplicative score rule, factoid weights $(0.70,0.20,0.10)$ give approximately $(0.434,0.134081,0.069628)$ for the three answering units. This toy restores the desired order; multiplication alone is not a general calibration method, particularly for negative scores. For a thematic query, weights $(0.10,0.20,0.70)$ make the section score $0.70\times0.617=0.432$. MoGG removes the requirement that levels be nested. It links chunks above a similarity threshold and uses graph hop range as granularity. Small-to-big is a useful first baseline: retrieve fine units and return their parents. It avoids cross-level raw-score comparisons. It can still miss thematic queries when no fine unit carries the broad selection signal; this is a workload risk, not an inevitable failure. When labels are missing, retrieve separately per level and use RRF. It fuses ranks rather than raw scales. Sparse scoring has the opposite length pathology. In tf-idf cosine, only query terms enter the numerator while the document norm grows. Long documents are over-penalized. Pivoted length normalization and BM25 address that sparse problem. Dense pooling has nonzero background contributions in both numerator and denominator, so its ratio flattens.

Worked clinical-reference example

The corpus has $20{,}000$ documents at $4{,}000$ tokens each, or $8\times10^{7}$ tokens. Dimension $d$ is 768 in fp32. Sentences average 25 tokens, paragraphs 100, and sections 400.

Take two extra searches at $2\,\mathrm{ms}$ each as a workload assumption, giving $4\,\mathrm{ms}$ if run serially. Parallelization, routing and candidate merging change wall time. Five $400$-token sections give $2{,}000$ prompt tokens. At $N=7\times10^9$ and assumed sustained throughput $1.5\times10^{14}\,\mathrm{FLOP/s}$, the $2NT$ proxy is $2.8\times10^{13}$ FLOPs or approximately $186.667\,\mathrm{ms}$. Five $25$-token sentences give $125$ tokens and approximately $11.667\,\mathrm{ms}$ under the same proxy. The prefill-proxy difference is $175\,\mathrm{ms}$. Dividing by the assumed $4\,\mathrm{ms}$ extra search work gives $43.75$, not a $44$-fold end-to-end speedup. If the route actually replaces sections with sentences, net idealized savings are $171\,\mathrm{ms}$ before router and merge overhead. DPR’s $21{,}015{,}324$ vectors at $d=768$ and four bytes each use $64.6$ GB, matching the reported approximately $65\,\mathrm{GB}$ flat index. Section vectors add $0.6144\,\mathrm{GB}$ payload and $0.0512\,\mathrm{GB}$ base-layer links; sentence vectors add $9.8304\,\mathrm{GB}$ payload and $0.8192\,\mathrm{GB}$ links. These budgets omit upper graph layers, text, IDs and runtime overhead.

Practical routing decisions

• Default to sentence and section levels. They total $3.4$ million vectors in the example, versus $4.2$ million with paragraphs added.
• Add another level only when query logs reveal a distinct answer-length mode.
• Ship small-to-big first. Consider a router when more than roughly one quarter of queries are thematic.
• Estimate $\rho$ from 200 same-document sentence pairs and $b$ from 200 query to non-answer sentence pairs.
• Treat the crossover as a check on the toy geometry, not an automatic router decision. Validate finite-level scores and task quality on the actual workload.
• Fail open to the middle granularity when router weights are near-uniform.
• Cap coarse contribution to the context budget because unusual queries can force expensive section prefills.
• Budget granularity against per-query prefill before one-time index memory. The example adds $11.3152\,\mathrm{GB}$ of payload-plus-base-layer capacity over the paragraph baseline and assumes $4\,\mathrm{ms}$ extra search work; level choice moves the prefill proxy by $175\,\mathrm{ms}$.

13.4 Propositions and fine-grained retrieval

Pooling dilution

• What: A passage contains $m$ atomic propositions with mean pairwise similarity $\rho$.
• Why: One exact fact can contribute only a fraction of a pooled passage vector.
• Failure without it: A broadly topical passage with no answer can outrank the passage that contains the exact answer.
• Cost and complexity: Proposition retrieval removes dilution by creating more vectors and requiring model-written, self-contained units. Assume $m$ unit proposition vectors, that the query equals one of them, and that its mean similarity to the other $m-1$ vectors equals the overall mean pairwise similarity $\rho$. The pooled gold-passage cosine is:

At $m=8$ and $\rho=0.5$, the score is $\sqrt{\frac{4.5}{8}}=0.75$. For a distractor with eight propositions, each at query similarity $\gamma=0.6$ and the same $\rho$, the score is $\frac{4.8}{6}=0.80$. The distractor wins whenever

That threshold is the square of the gold passage score. It is $0.5625$ in this example. At proposition granularity, the exact fact scores $1.00$ and the distractor scores $0.60$.

Why propositions are not sentence splits

• What: A proposition rewrites a fact so it survives without neighboring text.
• Why: A sentence like “the base filing fee rises to $\$780$ that year” lacks the visa class and year.
• Failure without it: Short sentence chunks can be precise but unsearchable because pronouns and references have no anchor.
• Cost and complexity: Rewriting requires a generation pass. Store offsets to the parent because the indexed proposition is model output, not original corpus text. The self-contained rewrite in the source is “The base H-1B filing fee for fiscal year 2026 is $\$780$.” Chen et al. (2024) compare passage, sentence, and proposition indexing across five open-domain question-answering datasets and six dense retrievers. Proposition indexing beats both alternatives, with the largest margin at small $k$. Sentence retrieval sits between passages and propositions. The claim is comparative, not a guarantee for every corpus. Fine granularity breaks in two directions.

1. Coverage falls when a question needs several facts. One passage may bundle them, while several propositions must all reach top $k$.
2. Sufficiency falls when a proposition loses conditions such as effective dates or employer-size qualifiers. Entity-relation triples go below the practical floor when they discard conditions. Stop at the smallest unit that is decidable on its own.

Index fine and serve coarse

• What: Search propositions, map hits to parent passages, deduplicate, then send parents to the generator.
• Why: Matching wants precision. Generation wants context.
• Failure without it: Serving only propositions strips qualifiers. Searching only passages restores pooling dilution.
• Cost and complexity: The design needs a parent map, source offsets, more index memory, and a measured fan-in. RAPTOR attacks the broad-query side by building a tree of recursive summaries. It composes with proposition retrieval rather than replacing it.

Worked policy corpus

The corpus has 2 million passages of 100 words. Dimension $d$ is 768 in fp32. A passage averages five 20-word sentences and about eight propositions. Rewriting expands total words to about $1.4\times$ the original.

Using the $187$ output tokens per eight-proposition passage budget below gives $187/8=23.375$ tokens per proposition on average. Thirty such propositions average about $701.25$ tokens, not exactly $690$, and may omit neighboring qualifiers. The $2{,}394$-token parent-text plan fits within $4{,}000$ tokens only if the remaining instructions, query, formatting and any reserved output fit in the remaining $1{,}606$-token budget. Decomposition uses 333 input tokens per passage. That contains 200 instruction and exemplar tokens plus 133 passage tokens. It produces 187 output tokens. Across 2 million passages, that is 666 million input tokens and 374 million output tokens. At $\$0.15$ per million input tokens and $\$0.60$ per million output tokens, the bill is $\$99.90+\$224.40=\$324.30$ once under these assumed prices. The added payload-plus-base-layer capacity is $53.248-6.656=46.592\,\mathrm{GB}$. This is recurring provisioned capacity, not a monthly dollar bill without a specified memory rental price. At $96\,\mathrm{B}$ per product-quantized code, the proposition payload becomes $1.536\,\mathrm{GB}$. Retaining the same base-layer neighbor budget leaves $1.536+4.096=5.632\,\mathrm{GB}$ before codebooks, upper layers, metadata, text and any full-precision rerank vectors; it is not a $1.5\,\mathrm{GB}$ all-in index.

When fine granularity pays

The gain is

At $m=8$, $\rho=0.2$, the idealized cosine-score headroom is $0.45227744$, about $45.23$ cosine points on a $0$–$100$ scale. It is not an accuracy gain. At $m=8$, $\rho=0.9$, the same cosine headroom is $0.04475134$, about $4.48$ cosine points, not a measured relative quality gain.

• Favor propositions for mixed reference material such as policies, frequently asked questions, catalogs, and long wiki pages.
• Skip them for tightly coherent prose with little pooling headroom.
• For 800-token parents, fan-in $1.7$ and $k=30$ produce $18\times800=14{,}400$ tokens. Serve a sentence window instead.
• Set $k$ as target parent count times measured fan-in. In the example, $18\times1.7$ is about 30.
• If fan-in is near $1.0$, the hits scatter across documents. Treat that as a multi-hop signal rather than merely increasing $k$.
• Reject propositions with unresolved pronouns, bare demonstratives, or missing named entities.
• The source cites contextual retrieval reporting a top-20 failure-rate reduction from $5.7\%$ to $3.7\%$ using a generated context prefix. It proposes that cheaper option for heavily anaphoric corpora.

13.5 Evaluating a chunker: $\mathop{\mathrm{IoU}}\nolimits$ and segmentation metrics

Measure token extent, not chunk identity

• What: Annotate each gold answer as raw source token offsets before running any chunker.
• Why: The same annotation remains valid across chunking policies.
• Failure without it: Chunk IDs become stale after every boundary change. End-to-end answer accuracy cannot attribute a change to the chunker.
• Cost and complexity: Human annotators must mark the actual answer excerpt. The example uses 200 queries. For query $q$, let $E_q$ be gold source positions and $T_q$ the union of retrieved source positions, keyed by document and token offset. Require nonempty gold for the recall formulas. If no text is retrieved, define recall and IoU as $0$ and declare a precision convention rather than dividing by zero. Duplicate prompt copies count once in $T_q$ but still cost prompt tokens. Recall

Precision

For nonempty gold and retrieved sets with positive intersection, let $a=\lvert E\cap T\rvert$, $e=\lvert E\rvert$, $t=\lvert T\rvert$. The reciprocal identity below needs $a\gt 0$; the direct $F_1$ identity also extends to zero overlap when $e+t\gt 0$.

Per query, IoU is a monotone transform of $F_1$ and adds no ranking information. Averaging this nonlinear transform across queries can change system rankings, so macro-mean IoU is not the transform of macro-mean $F_1$. $F_1=0.80$ gives $\mathop{\mathrm{IoU}}\nolimits=2/3\approx0.667$; $F_1=0.50$ gives $\mathop{\mathrm{IoU}}\nolimits=1/3\approx0.333$. Returning the whole gold-containing document gives token recall $1$ conditional on retrieving that document. For $3{,}000$ source tokens and a $60$-token gold excerpt, precision and IoU are $60/3{,}000=0.02$. A one-chunk-per-document index does not by itself guarantee the right document is retrieved. Embedding similarity cannot replace extent accounting. A 60-token answer plus 500 tokens of on-topic padding can remain strongly similar to the gold excerpt.

The three extent errors

For gold tokens 81 through 120 in a 200-token document:

$\mathop{\mathrm{IoU}}\nolimits$ cannot distinguish the direction of error by itself. Too-small and misaligned both score $0.50$. Report precision and recall beside it.

Boundary-only metrics

• What: $P_k$ checks whether endpoints at separation $k$ agree on same-segment membership in reference and predicted segmentations.
• Why: It evaluates boundaries independently of retrieval and an encoder.
• Failure without it: Retrieval IoU also reflects candidate selection, not only segmentation.
• Cost and complexity: $P_k$ can treat missing and extra nearby boundaries differently; WindowDiff instead checks boundary counts. Both depend on window conventions and reference segment lengths. WindowDiff compares the number of boundaries inside the window rather than a same-segment bit. For uniform reference segments of length $\ell$, the conventional window is $k=\frac{\ell}{2}$. For long uniform reference segments of even length $\ell$, $k=\ell/2$ gives an approximately $0.5$ no-boundary baseline for both metrics. The all-boundary baseline is approximately $0.5$ for $P_k$, but approximately $1$ for WindowDiff when $k\gt 1$ and every predicted window has more boundaries than the reference. Finite endpoints and rounding affect the values; compute both baselines on the actual reference. The chapter traces element-level precision, recall, and overlap to Extensible Markup Language (XML) retrieval, where systems had to return the correct section, paragraph, or span from a nested document.

Worked chunk-size comparison

The illustrative evaluation has $200$ queries with exactly $60$ gold tokens each, not merely a mean of $60$. Starts are uniform modulo the chunk length. Assume an oracle returns the one chunk with greatest gold overlap; these are idealized top-one extent metrics, not measured retriever performance. For chunk length $c$ and overlap $o$, stride is $c-o$. Split probability is

For no overlap, a continuously uniform split fraction gives conditional best-piece recall $3/4$. With an integer $60$-token span, the $59$ possible interior cuts give $\sum_{j=1}^{59}\max(j,60-j)/(59\times60)\approx0.745763$ instead. Use integer phases for the table below. For a uniform split fraction $u$, the expectation is

The large chunk wins oracle top-one recall by $6$ percentage points. The ratio of mean per-query IoUs is approximately $4.6285$. Neither $\mathbb E[\mathrm{IoU}]$ nor a five-chunk retrieval metric can be inferred by applying IoU to a rounded mean overlap; the prompt column is a separate $k=5$ cost plan. At $10{,}000$ queries per day and $\$3$ per million input tokens, the prompt costs are $\$30$ and $\$150$ per day. The difference is $\$120$ per day or $\$43{,}800$ per year. At $c=200,o=60$, every fitting $60$-token span is intact because $o\ge59$. The long-document overlap multiplier is $200/140\approx1.428571$, about $42.857\%$ extra unpadded tokens; finite vector and graph inflation still depend on document lengths. A 1,000-token chunk with 200-token overlap also protects the 60-token excerpt. A 200-token chunk with 40-token overlap has split probability $\frac{59-40}{160}=0.11875\approx0.119$.

Evaluation decisions

• If source edits make offsets stale, anchor each annotation to a quoted string and relocate it at evaluation time.
• Compare chunkers at a fixed retrieved-token budget, not fixed $k$.
• If the unique-source-position budget is exactly $\lvert T\rvert=B$, precision is $R\lvert E\rvert/B$. A fixed billed prompt-token budget need not fix this denominator when chunks overlap or duplicate; retrieval and encoder effects still influence coverage.
• If the reranker is the binding limit, fix candidate count instead.
• Set overlap from the 95th-percentile annotated excerpt length.
• If $1.43\times$ index inflation is too high, accept a measured split rate and explicitly retrieve the adjacent chunk or parent. Merely raising $k$ to $2$ does not guarantee that neighbor is returned.
• Use WindowDiff when reference boundaries cluster.
• A reported $\mathop{\mathrm{recall}}\nolimits @5$ rise from $0.71$ to $0.79$ is not enough when mean chunk length also rises from 220 to 610 tokens. That is $2.8\times$ more text and $3{,}050$ prompt tokens at $k=5$ for a 60-token answer.

13.6 Does semantic chunking actually pay?

Semantic chunking as detection

• What: Embed sentences $e_1$ through $e_n$ and compute adjacent distance $d_i=1-\cos(e_i,e_{i+1})$.
• Why: A large $d_i$ may indicate a topic boundary.
• Failure without it: A claimed boundary can look almost identical to an ordinary within-topic transition in one coherent document.
• Cost and complexity: The detector needs a full sentence-embedding pass before the final index pass. As a local approximation, model within-topic cosine gaps as $\mathcal N(\mu_w,\sigma^2)$ and true-boundary gaps as $\mathcal N(\mu_b,\sigma^2)$. Cosine distance actually lies in $[0,2]$; the Gaussian approximation ignores its tails outside that interval. The equal-variance assumption is part of the example. Different variances, prevalence or score calibration change the detector’s precision–recall curve. Separability

The source fixes $\sigma=0.08$ for its worked model. For stitched text, $(0.90-0.25)/0.08=8.125$. Inside one 40-page insurance policy, $\mu_b=0.35$ and $\mu_w=0.25$. $\Delta=1.25$. The ratio of the two standardized mean separations is $8.125/1.25=6.5$. This compares model inputs, not an independently measured transfer penalty.

Same threshold, two corpora

Take a 200-sentence article with 8 real boundaries among 199 gaps. Boundaries are $4.0\%$ of gaps. A strict empirical $95$th-percentile threshold yields roughly $10$ cuts among $199$ distinct gaps, with details set by quantile and tie conventions. Population-mixture thresholds below approximate this top-$5\%$ rule; a finite document need not realize the expected count. For threshold $t$, boundary recall is

FPR is

Qu, Tu, and Bao (2024) report no consistent performance gains justifying semantic chunking’s cost in their evaluated tasks and corpora. The Gaussian example illustrates one possible mechanism; it does not establish that the paper’s negative result holds exactly below a particular $\Delta$. Heterogeneous logs or bundles may create clearer boundaries, but validate the benefit on real documents.

Overlap as the cheaper repair

For answer span length $L = 80$, chunk length $C = 512$, and stride $S = C-o$, intact probability is

At $o=0$, intact probability is $\frac{433}{512}=84.6\%$. Split rate is $15.4\%$. At $o=50$, intact probability is $\frac{433}{462}=93.7\%$. Split rate is $6.3\%$. At $o=50$, split probability falls from $79/512\approx15.4297\%$ to $29/462\approx6.2771\%$, a reduction of about $9.1526$ percentage points. Long-document token inflation is $C/S=512/462\approx1.108225$, or $10.8225\%$ extra; the multiplier itself is not $10.8\%$. Semantic output has a length distribution. A coherent run of 90 sentences at about 25 tokens each becomes roughly $2{,}250$ tokens. A 512-token encoder indexes less than one quarter and silently drops more than three quarters unless the run is capped.

Worked ingestion comparison

The corpus has $1.2$ million articles at $3{,}000$ tokens each. $T$ is $3.6\times10^{9}$ tokens. At 25 tokens per sentence, it contains $1.44\times10^{8}$ sentences. The encoder has 110 million parameters and costs $2N=2.2\times10^{8}$ FLOPs per token. Sustained throughput is $3.4\times10^{14}\,\mathrm{FLOP}/\mathrm{s}$. Compute costs $\$2.50$ per graphics processing unit (GPU) hour.

Sentence batches pad from a 25-token mean to 32 token slots. That creates the $4.61\times10^{9}$ detection slots. All-pairs dot products for $120$ normalized $768$-dimensional sentence vectors require $\binom{120}{2}=7{,}140$ pairs and about $1.536\times10^3$ FLOPs per pair. Corpus work is $1.3160448\times10^{13}$ FLOPs, a $0.0387072\,\mathrm s$ arithmetic proxy. This excludes clustering, storage, kernel launches and orchestration; it is not measured total clustering time. Including final embeddings on both sides, the agentic proxy is about $33.9314$ times semantic ingestion compute. Generation alone costs about $\$123.53$; total with final embeddings is about $\$125.15$. Bandwidth-bound serial decoding can make actual time substantially larger. A random detector placing 10 cuts among 199 gaps with 8 true boundaries has expected precision $\frac{8}{199}=4\%$. The modeled $24\%$ is six times better than chance and still puts three cuts in four inside a topic.