Fixed-size chunking fragments discourse [15]. Structure-aligned alternatives can outperform it by large margins. A 36-method, 6-domain, 5-embedder comparison reports paragraph-group chunking at mean nDCG@5 \( \approx\) 0.46, versus < 0.25 for fixed character windows [3]. But the gains are unstable. [2] [2] — the keystone non-replication result — find semantic chunking’s advantages inconsistent across document retrieval, evidence retrieval, and generation, and not worth the compute. A follow-up analysis attributes the measured gains chiefly to synthetic corpora stitched from unrelated documents [9]. Chunk-size effects also interact with task and embedder, and they flip sign between models [13]. This instability motivates our design rather than obstructing it. We hold corpus, embedder, and chunk size fixed, and we vary boundary placement alone (Section 3.2).

Anthropic’s Contextual Retrieval [4] prepends a 50–100-token LLM-generated situating context to each chunk. It reports 35–49% reductions in top-20 retrieval failure. Those figures come from an internal, non-peer-reviewed benchmark, which is itself part of the gap we address. Architectural variants reach the same goal without generation. Late chunking pools token embeddings after a full-document encoder pass [5]. Contextual document embeddings train the encoder to represent neighbors [6]. The ConTEB benchmark and InSeNT training target the setting directly [10]. Yet added context is not reliably helpful. Late chunking drops 21 nDCG@10 on CovidQA [10]. Anthropic-style contexts produce only marginal gains in independent evaluation [15]. Contextualization helps in-corpus retrieval while it hurts in-document retrieval [9]. Our A1 arm implements the Anthropic recipe verbatim: per-chunk LLM contexts on fixed windows. This fragile mechanism therefore competes directly against boundary placement and metadata injection under one metric.

Production RAG systems commonly prepend a chunk’s native heading path — the document title and section trail, essentially a breadcrumb. But we could not verify any study that isolates this practice as its own treatment, separate from boundary placement and from LLM-generated context. It is the least-studied of the four mechanisms. It is also the site of our strongest novelty claims: the isolated measurement (A2 vs. A3) and the placebo control (A3 vs. P) that separates path content from token injection.

RAPTOR builds a synthetic summary tree by recursive clustering and retrieves across its levels [7]. Note that RAPTOR constructs the tree — it does not use the document’s native hierarchy. Our G-conditions close that scope difference with real heading trees. Two-stage retrieval helps in some settings. Dense Hierarchical Retrieval reports up to +12% absolute top-1 over flat DPR [8]. But strong recent results go the other way. A flat retrieve-then-read baseline that preserves document order matches or beats multi-stage systems [11]. Topology-aware analyses find that RAPTOR-style semantics-first retrieval loses to flat baselines on structural tasks [12]. The hierarchy that does help is parent–child reranking, not route-to-section [16]. Our A4/G4 arms contribute a controlled data point to this debate. A diagnostic (first-stage section recall) localizes why routing loses (Section 6.4).

Coverage- and diversity-aware effectiveness measures descend from \( \alpha\) -nDCG [14], which scores novel information rather than repeated relevance. Our coverage-nDCG applies the same principle at the character-span level, where one question’s evidence spans several chunks (Section 5.1). Placebo-style null controls are standard in medicine, and researchers increasingly urge them for ML ablations. None of the structure/context retrieval studies above includes one. Finally, non-replication is documented within this subfield [2, 9],. We respond in kind: replication across corpora is part of the result, not future work.

All conditions share one corpus, one embedder per run, and one retrieval budget. They differ only in (a) where chunk boundaries fall, (b) what prefix precedes the content at embedding time, and (c) whether retrieval is flat or two-stage. Chunks are contiguous, non-overlapping spans that exactly tile each document. The prefix, when present, renders as [Title › Section › … ] \n\n content. Prefix tokens are tracked separately from content tokens throughout.

Figure 1 shows what each mechanism does to a document at embedding and retrieval time.

Pan/Zoom

We fixed four contrasts in the configuration before the runs. Each contrast isolates one mechanism question:

• C1 (organization vs. contextualization), A3-A1: do structure-aligned boundaries plus native metadata beat the strongest generative treatment on fixed windows?
• C2 (content vs. tokens), A3-P: does the content of the heading path matter, or is the gain token injection?
• C3 (cost of induction), G3-A3: how much is lost when an LLM induces the structure instead of the author?
• C4 (routing vs. flat), A4-A3: does two-stage hierarchical retrieval beat flat search over the identical index?

Table 4 in the appendix lists the experimental conditions employed in this study.

Size matching (guards C1). The fixed-window baseline is not cut at an arbitrary size. Per document, its target window equals the mean token length of that document’s induced chunks. Realized means differ by \( \leq 4\) tokens per chunkset (Wikipedia-FA: fixed 255.4 vs. induced 259.6 content tokens; QASPER: 244.9 vs. 249.8). The comparison of A0/A1 vs. A2/A3 therefore isolates boundary placement, not chunk size. Window edges snap to sentence boundaries. Structural chunks split (at sentence boundaries) or merge when they exceed 350 or fall below 60 content tokens.

Placebo construction (guards C2). Placebo paths are real heading paths from other documents. We bucket unique (document, path) pairs by depth and permute them within-bucket toward a cross-document derangement (fixed seed). We count and report residual same-document assignments. Placebo prefixes therefore match real breadcrumbs in format, depth, and length distribution, while they carry no information about the chunk. This is a semantically null treatment in the strict sense.

Representation constancy (guards C4). The two-stage conditions build section centroids as the renormalized mean of the same chunk vectors that flat retrieval searches. C4 therefore compares retrieval algorithms, never representations. Stage 1 ranks section centroids and keeps the top 5 (a section = document + top-level heading). Stage 2 ranks chunks within the routed sections. If fewer than k candidates remain, the system backfills from the global ranking. A per-query section-recall diagnostic records whether stage 1 retained any section that overlaps the gold evidence. This separates routing failures from ranking failures.

Leakage and provenance controls. The pipeline selects the gold evidence spans. The query-writing LLM never self-reports them. Synthetic queries pass a two-round lexical-leakage filter (Section 5.1). Relevance is computed purely from character-span overlap against content-only offsets, so prefixes can never create spurious relevance.

The induced-structure conditions (A2–A4, P) use a hierarchy from a local Gemma-4-26B-A4B model (the Mixture-of-Experts variant) [17]. The model sees the document’s paragraphs with all headings stripped. It returns sections as (title, level, paragraph-range) tuples under hard constraints: sections tile the paragraphs exactly, and levels increase by at most one. We verify the constraints mechanically. A failure gets one self-correction round with the specific violations, then a deterministic repair that snaps near-valid output to a legal tiling. On Wikipedia-FA, 198/200 documents produced usable structure (116 required repair; mean 21.9 sections/doc). On QASPER, 1,579/1,585 did (590 repaired; mean 14.5 sections/doc vs. 14.8 gold). One fact is critical for C3: the inducer does not simply recover native sectioning. Only 78.9% (Wikipedia-FA) and 75.8% (QASPER) of induced chunk spans are byte-identical to a gold span. Gold and induced conditions therefore measure genuinely different segmentations.

We sample documents from Wikipedia’s Featured Articles under quality filters (\( \geq 3,000\) words, heading depth \( \geq 3\) , \( \geq 12\) sections) with topic-bucket balancing We classify each article into one of six keyword-defined topic buckets (history/war, biography, science/nature, arts/media, geography/places, other) from its Wikipedia categories, and cap each bucket at 60 documents during seeded sampling (seed 20260710). The cap binds for two buckets; the realized corpus is history/war 60, other 60, biography 30, arts/media 22, science/nature 15, geography/places 13. The corpus remains topic-skewed, reflecting real Featured-Article imbalance; the cap limits rather than eliminates it.. A single canonical normalizer flattens the wikitext. Gold structure is the article’s own heading tree, offset-verified.

Queries are LLM-generated but provenance- and leakage-controlled. For each document, the pipeline itself samples gold spans and requests four query types. A factoid query targets one span. An underspecified query targets one span but names the referent only vaguely — "the commander," never the proper name. A multihop query targets two spans in different top-level sections. A structural query targets a whole top-level section. Each generated query must pass a lexical-leakage filter: reject if ROUGE-L F1 vs. its gold span \( > 0.35\) or if any verbatim 4-gram is shared. A second LLM call then paraphrases the query. The paraphrase — which is what enters the benchmark — must pass the same filter again. The funnel: 2,306 generated → 1,945 survivors → 951 balanced finals (250 factoid, 250 underspecified, 250 structural, 201 multihop) over 184 documents. Shipped queries are lexically far from their evidence (typical ROUGE-L 0.02–0.13). BM25 places the gold chunk first for only 36.3% of queries, so the benchmark is not trivially lexical.

QASPER [18] provides NLP papers with questions from real readers and gold evidence paragraphs — human-authored relevance on a genre far from encyclopedic prose. We merge all splits (train/val/test), because the study is an Information Retrieval (IR) ablation, not supervised learning. Gold structure is the paper’s own section tree. We harvest questions instead of generating them. We match each question’s evidence paragraphs to the flattened document (exact → fuzzy → section fallback; 95.7% exact). This yields 5,049 → 4,303 usable questions over 1,508 papers, after we drop unanswerable, no-evidence, and numeric-only items. The set is hard and non-lexical (BM25 gold-top-1 22.7%). It is also genuinely multi-span: 43% of questions cite multiple evidence paragraphs (mean 2.10 spans/question). That is precisely the regime coverage-nDCG exists for.

One structural property matters when we interpret C3. QASPER’s native hierarchy is shallow: 61% of papers have depth-1 section lists, versus depth \( \geq 3\) for every Wikipedia-FA document by construction. Gold and induced trees therefore have less room to diverge (Section 6.5).

Coverage-nDCG@10 (primary). Walking down the ranking, a chunk’s gain is the fraction of gold-evidence characters it covers that earlier chunks did not already cover. The DCG of these marginal gains is normalized by the DCG of the ideal ordering achievable from that condition’s own chunk set. The novelty term prevents small-chunk conditions from banking the same span repeatedly. Per-condition normalization scores each condition against the best it could have done with its own segmentation, so a condition is not penalized merely for a different chunking. Both properties are required for fair comparison across chunkings. This is why binary nDCG is insufficient here.

Secondary metrics form a ladder from rank-insensitive to application-realistic: recall@50 over gold spans (\( \geq 50\%\) covered by the union of top-50), Mean Reciprocal Rank (MRR), binary nDCG@10, and token-budget coverage at 512/1024/2048 tokens. In the token-budget metrics, prefix tokens count against the budget. Each treatment is charged its real context-window cost.

Relevance rule. A chunk is relevant to a span if and only if it covers \( \geq 50\%\) of the span, or \( \geq 80\%\) of the chunk lies inside the span. All overlap is computed on content-only character offsets.

We test all four contrasts on one cell fixed in advance — cov-nDCG@10, strong embedder, no reranker — to prevent metric shopping. Each contrast is the unweighted mean of paired per-query deltas. Multiple queries share a document, so we estimate uncertainty with a cluster bootstrap that resamples documents with replacement (10,000 iterations) [19]. P-values are two-sided and Holm-corrected across the four contrasts [20]. Effect sizes are Cohen’s \( dz\) [21]. Everything outside the primary cell is explicitly exploratory, and Section 6 labels it as such. This includes per-condition means across metrics and cells, the breadcrumb-depth dose-response, and per-query-type breakdowns.

We report a mixed-effects regression (score   condition, random intercept per document) as a secondary check. We flag its specification honestly. It omits a per-query random effect although treatment varies within query. Query-level variance therefore inflates its standard errors, and the model is conservative for small paired effects. We report it because pre-registration said we would. Discordances are discussed where they occur (Section 6.1).

Embedders are bge-small-en-v1.5 ("weak") and bge-large-en-v1.5 ("strong") [22]. Every run uses exactly one embedder. The pair exists to test robustness across encoder capacity, not as a two-stage system. The reranked cells apply bge-reranker-v2-m3 [23] over the top 50 candidates of the strong embedder. All generative stages (structure induction, A1 contexts, query generation and paraphrase) ran on local Gemma-4-26B-A4B via vLLM [24] on a single H200 GPU. Generation is confined to one-time corpus and query construction under fixed seeds with on-disk caching. The measured half of the pipeline — embedding, retrieval, scoring, statistics — is deterministic. Code, configurations, and per-query results will be released.

Table 1 reports the primary cell. Three contrasts replicate across both corpora with the same sign and Holm significance. One is corpus-dependent.

On Wikipedia-FA the secondary MixedLM flags only C4. This is the expected signature of its specification, not evidence that the other effects are fragile. The model lacks a query random effect, so it absorbs large query-to-query variance into its residual. It can therefore detect only the largest effect (\( |dz| = 0.17\) , roughly twice the others). At QASPER’s \( n = 4,303\) queries, even this conservative model confirms all three non-null contrasts. The bootstrap is correctly specified for a paired, document-clustered design, and it is primary throughout.

Effect sizes are small: +0.01-0.02 cov-nDCG@10, \( dz\) 0.06-0.11. We ask the reader to hold both halves of that sentence. The effects are small, but they are also real: given tight CIs, Holm correction, and same-sign replication across two genres and two query provenances. Steady gains of this size are within the range IR practice treats as meaningful. In a subfield documented for non-replication, the contrasts’ cross-corpus consistency is itself a finding.

Table 2 shows per-condition means decompose the organization effect (strong embedder, no reranker):

Two regularities hold on both corpora. First, the boundary step dominates the metadata step. A0→A2 — a re-cut of the same text at induced structural boundaries, with no prefix — is worth +0.041 on Wikipedia-FA and +0.019 on QASPER. A2→A3, the addition of the real heading path, adds only +0.011 and +0.009. Second, A1 \( <\) A2. LLM-written per-chunk contexts on fixed windows lose to bare structural cutting with no prefix at all. The generative treatment does not recover what bad boundaries destroy. Both regularities also hold on the weak embedder (Wikipedia-FA A0 0.306 → A3 0.343; QASPER 0.087 → 0.107). The ordering is therefore not an artifact of encoder capacity.

Contextualization’s cost becomes explicit under token budgets, where prefixes are charged to the context window. At a 512-token budget, A1 falls below plain A0 on both corpora (Wikipedia-FA 0.167 vs. 0.187; QASPER 0.046 vs. 0.052). A1 never clearly recovers by 2,048 tokens (0.396 vs. 0.397). An 80-token generated context that mostly restates document-level information displaces retrievable evidence.

The placebo arm exists to answer one question: is the A2→A3 gain merely "more tokens move the embedding somewhere useful"? On both corpora the answer is no. Real paths beat shuffled paths of identical format, depth, and length (+0.0099 Wikipedia-FA; +0.0161 QASPER, where C2 is the largest and most significant contrast). The placebo’s position against the bare structural arm tells the same story. Shuffled paths buy nothing over no prefix at all (Wikipedia-FA: P 0.403 vs. A2 0.403), or they actively hurt (QASPER: P 0.109 vs. A2 0.117). The full QASPER ordering is instructive: A0 0.098 \( <\) P 0.109 \( <\) A1 0.114 \( <\) A2 0.117 \( <\) A3 0.125. Null tokens buy a little over raw windows. Contextualization buys more. Bare structural cutting buys more still. Real metadata buys the most.

Without arm P, none of this would be attributable. A gain from any added prefix could be an embedding-geometry artifact, and the A2→A3 step would remain ambiguous between content and token injection. With arm P, the attribution is measured. The prefix’s value depends on its being true of the chunk. Semantically null tokens are at best neutral and at worst noise.

C4 is negative on both corpora. It is the most robust single effect in the study (Wikipedia-FA \( dz = -0.17\) ). The two-stage conditions retrieve over the identical vectors as A3, so the penalty is attributable entirely to routing. The section-recall diagnostic localizes it. The probability that stage 1 even retains a gold-bearing section is only 0.60–0.68 on the 200-document Wikipedia-FA corpus. It collapses to 0.17–0.20 on QASPER, where the router faces \( \approx 22,000\) candidate sections. Whatever stage 1 drops, stage 2 cannot recover. Recall@50 shows the same shadow (QASPER: A4 0.293 vs. A3 0.302; Wikipedia-FA: 0.675 vs. 0.730). The penalty scales with the routing failure. Proportionally, the loss is \( \approx8\%\) of A3’s score on Wikipedia-FA and \( \approx12\%\) on QASPER — larger where section recall is worse — even though QASPER’s absolute delta is smaller (its whole scoreboard is \( \approx 3 \times\) lower).

One qualification keeps this negative honest: it is largely a no-reranker phenomenon. Under bge-reranker-v2-m3, the QASPER gap shrinks from -0.0147 to -0.0038 (Wikipedia-FA: -0.0326 → -0.0273). The reranker re-scores a wide flat candidate list that routing would otherwise have pruned. The correct reading is therefore narrow. Route-to-section-then-rank under-recalls as the section inventory grows. It loses to flat search over the same index unless a reranker restores the candidates. This is a result about one widely-used naive architecture, not about hierarchy in general. It is consistent with the literature’s split verdict [11, 8, 16].

On Wikipedia-FA, native heading trees beat Gemma’s induced trees by +0.0129 (Holm-significant). Imperfect induction has a small, measurable cost. On QASPER the advantage vanishes (+0.0011, n.s.). This is not because the inducer copied the papers’ sectioning: span identity is 75.8%, comparable to Wikipedia’s 78.9%. Both readings are findings. For the gold-structure story: the advantage of native structure does not generalize across genres, and no one should claim it corpus-generally. For the induction story: a 26B open-weights model re-segments a quarter of all spans and loses nothing measurable against author-written structure on scientific papers. A plausible mechanism — untested, offered as hypothesis — is depth headroom. 61% of QASPER papers are depth-1, so gold and induced trees have little room to differ in the information the breadcrumb carries. Wikipedia-FA’s \( \geq3\) -deep trees give gold structure more leverage.

Breadcrumb depth dose-response. We regress cov-nDCG@10 on injected path depth (0→3, i.e. A2 → A3-d1 → A3-d2 → A3). The strong embedder shows a small positive slope on both corpora (Wikipedia-FA +0.0032/level, bootstrap p = 0.02; QASPER +0.0027/level, p < 0.001). The weak embedder is flat on Wikipedia-FA (+0.0006/level, p = 0.65) and marginal on QASPER (+0.0012/level, p = 0.052). In every cell, most of the benefit arrives at the first path level and then plateaus (e.g., Wikipedia-FA strong: 0.403 → 0.412 → 0.413 → 0.413). We do not promote this to a finding. It sits outside the pre-registered family. The weak-embedder slopes disagree across corpora. QASPER’s shallow native depth also partly collapses the depth arms by construction.

Per-query-type cuts. On Wikipedia-FA, the C1 gain concentrates on factoid (+0.046) and multihop (+0.031) queries. It is absent on structural queries (-0.002). C2 is largest on structural queries (+0.023). On QASPER, C1, C2, and C4 hold their signs across all three native question types (extractive/abstractive/boolean). The two corpora use incommensurable query taxonomies, so no specific type interaction can be called replicated. We report these cuts as hypothesis-generating only.

Reranking lifts all boats. A cross-encoder adds +0.04-0.06 cov-nDCG to essentially every condition on both corpora. It leaves the organization contrasts intact (Wikipedia-FA A3: 0.413 → 0.474; QASPER: 0.125 → 0.165). It disproportionately rescues only A4 (Section 6.4).

Recall, not just ranking. The contrasts partly live in recall. On QASPER, strong-embedder recall@50 spans 0.248 (A0) to 0.302 (A3). Structure-aligned chunking retrieves more of the evidence, not merely in better order. Absolute recall (\( \approx0.25–0.30\) ) shows this benchmark is far from saturated.

Every condition is one point in a design that varies exactly three things. First, where the chunk boundaries fall. Second, what prefix precedes the chunk content at embedding time. Third, whether retrieval is flat or two-stage. Chunks are contiguous, non-overlapping spans that tile each document exactly. One run holds all other factors fixed: corpus, embedder, and retrieval budget.

Condition names encode two axes. The letter encodes the source of structure:

• A means induced structure. The local Gemma-4-26B-A4B model produces the section tree after the pipeline strips the native headings (See appendix H).
• G means gold structure. The section tree is the document’s own author-written heading tree, offset-verified (See appendix E.2).

The number encodes the treatment level. The numbering stays parallel between the two families. A G-condition is therefore the exact counterpart of the A-condition with the same number. The two differ only in gold structure versus induced structure:

There is no G0 or G1, by design. Levels 0 and 1 use fixed windows and therefore have no structure. "Gold versus induced structure" is undefined for them. The G family exists only where a structural tree is in use (levels 2 to 4).

Two conditions extend the induced family for control purposes:

• P (placebo) matches A3 (induced structure, depth-3 heading-path prefix, flat). It differs in one way: each chunk’s prefix is a real heading path from a different document. The prefix matches the real ones in format, depth, and length distribution. It carries no information about the chunk. P isolates the token-injection confound (main text 3.2; construction in E.3 here).
• A3-d1 / A3-d2 / A3 form the breadcrumb depth ladder. The -dN suffix means "heading path cut to depth N." The bare name A3 is the full-depth (depth-3) rung. The "3" in "A3" is the treatment-level index. It coincides with the depth, which is why the name is not A3-d3. Read as a dose-response series, the ladder is A2 (depth 0, no prefix), then A3-d1, then A3-d2, then A3 (depth 3). The prefix renders as [Title › Section › … ] \n\n content. The separator is ›.

Each pre-registered contrast is a single subtraction that isolates one mechanism:

A cell is one fixed combination of the non-condition factors: one metric, one embedder, one reranker setting, one corpus. The full experiment is a grid over those factors. The conditions above re-run in each cell. One pre-registered primary cell gives the four contrasts their confirmatory test, which forecloses metric-shopping: cov-nDCG@10 × strong embedder (bge-large) × no reranker. Every other cell is exploratory, and main-text 6 labels it as such. This covers other metrics, the weak embedder, and the reranked variants. The grid axes are these. Metric is one of cov-nDCG@10, recall@50, MRR, binary-nDCG@10, or token-budget coverage at 512/1024/2048. Embedder is the weak bge-small-en-v1.5 or the strong bge-large-en-v1.5. Reranker is none or bge-reranker-v2-m3 over the top 50. Corpus is Wikipedia-FA or QASPER.

The evidence for a single query often spreads across several disjoint character spans. On QASPER, 43.9% of questions cite multiple evidence paragraphs (mean 2.10 spans/question). The primary metric therefore needs two properties. Binary nDCG provides neither:

1. Novelty (coverage) awareness. A chunk that repeats evidence already retrieved higher in the ranking should earn little. Credit accrues for new gold coverage only. Without this property, a condition that cuts text into many small chunks banks the same span again and again.
2. Per-condition ideal normalization. A condition is scored against the best ranking that its own chunk set allows. A condition is therefore not penalized for segmenting the document differently from another condition.

Coverage-nDCG applies the novelty principle of \( \alpha\) -nDCG [14] one level below sub-topic diversity. It applies it at the level of gold-evidence characters within a single query.

Let a query belong to document \( d\) with gold-evidence spans \( G = \{g_1, \dots, g_S\}\) . Each \( g_j = [\text{start}_j, \text{end}_j)\) is a half-open character interval on \( d\) ’s flat text, and \( |g_j|\) is its length. A retrieval run returns a ranked list of chunks \( c_1, c_2, \dots\) . A chunk from any document other than \( d\) contributes zero, because offsets are per-document coordinate spaces.

For each gold span \( g_j\) , keep the set of its characters already covered by chunks ranked above the current one. For the chunk \( c_i\) at rank \( i\) , the marginal gain is the mean over gold spans of the fraction of newly covered characters:

Here \( \text{covered}_{<i}(g_j)\) is the union of the portions of \( g_j\) covered by \( c_1, \dots, c_{i-1}\) . After the metric scores \( c_i\) , it adds the overlap of \( c_i\) with each \( g_j\) to \( \text{covered}(g_j)\) .

The discounted cumulative gain at cutoff \( k\) uses the standard log discount:

The ideal DCG runs over the same condition’s own chunk set by a greedy maximum-marginal-coverage ordering. It repeatedly appends the not-yet-selected chunk with the largest current marginal gain, up to \( k\) chunks. This greedy order is the standard tractable stand-in for the ideal ranking under a coverage objective. Finally,

The value is 0 when \( \text{IDCG@}k = 0\) . That case means no chunk of the condition covers any gold character.

Suppose one query has a single gold span \( g_1\) of length 100 characters. Condition X returns, in rank order, chunks that cover characters \( [0,60)\) , then \( [40,100)\) , then an irrelevant chunk:

• Rank 1 covers 60 new chars. The gain is \( 60/100 = 0.60\) . The discount is \( \log_2 2 = 1\) .
• Rank 2 covers \( [40,100)\) , but \( [40,60)\) is already covered, so 40 new chars. The gain is \( 0.40\) . The discount is \( \log_2 3 \approx 1.585\) .
• Rank 3 covers 0 new chars. The gain is 0.

The result is \( \text{DCG} = 0.60/1 + 0.40/1.585 = 0.852\) . The greedy ideal from this chunk set places the same two chunks first, because their gains are identical. So \( \text{IDCG} = 0.852\) and \( \text{cov-nDCG} = 1.0\) . Suppose instead a third chunk covered \( [0,100)\) in one shot. The ideal would rank it first (gain 1.0 at rank 1), and the observed ranking would score below 1.0. The example shows the two active ingredients. The rank-2 chunk is discounted for redundancy (40 new chars, not 60). The score is relative to the best that the condition’s own chunks allow.

Each contrast runs on paired per-query deltas. For every query answered under both conditions, \( \delta_q = \text{score}_A(q) - \text{score}_B(q)\) . The reported effect is the unweighted mean \( \bar\delta\) over queries. Pairing is the correct design, because every query is scored under both arms. Pairing removes the large query-to-query difficulty variance from the comparison.

Queries are not independent. Several queries share a document and move together. The protocol therefore estimates uncertainty with a document-clustered bootstrap, not an ordinary per-query bootstrap. The difference is the resampling unit:

• Group the per-query deltas by their document.
• For each of \( B=10,000\) iterations, resample documents with replacement. Every query of a drawn document enters the replicate as a bundle.
• Recompute the mean delta on each replicate: \( \bar\delta^{(b)} = \big(\sum_{d \in \text{draw}} \text{sum}_d\big) / \big(\sum_{d \in \text{draw}} \text{count}_d\big)\) .

The bootstrap distribution gives three quantities:

• The 95% CI is the 2.5th and 97.5th percentiles of \( \{\bar\delta^{(b)}\}\) .
• The two-sided p-value is \( 2 \cdot \min\big(\Pr[\bar\delta^{(b)} \le 0], \Pr[\bar\delta^{(b)} \ge 0]\big)\) , clipped to 1.
• The **effect size** is Cohen’s \( dz = \bar\delta / \operatorname{sd}(\delta_q)\) . The standard deviation runs over the per-query deltas (ddof = 1).

Resampling documents rather than queries widens the intervals. The wider intervals reflect the true number of independent units, which is documents, not queries. Take QASPER’s design: 4,303 queries over 1,508 documents. An ordinary bootstrap would treat the effective sample size as about \( 3\times\) too large and understate uncertainty. The cluster bootstrap is the correctly specified test, and it is primary throughout.

The protocol tests four contrasts as a family. It therefore adjusts the per-test p-values by the Holm step-down procedure. Sort the four p-values in ascending order. Inflate the \( i\) -th smallest (0-indexed) by the factor \( (m-i)\) with \( m=4\) , under a running maximum so the adjusted values stay monotone. Clip to 1. Holm controls the family-wise error rate at 0.05. It rejects more true effects than the blunt Bonferroni factor of 4.

The protocol reports a mixed-effects regression as a pre-registered secondary check. This model is misspecified for the design, by design, and the paper discloses this. The study is paired within query, because the same query appears under both conditions. But the model has no per-query random effect. The large query-to-query baseline variance therefore lands in the residual and inflates the standard error of the treatment coefficient. The model is therefore conservative for small paired effects. It resolves only the largest effect. This matches the observed pattern. On Wikipedia-FA (\( n \approx 951\) per condition) it flags only C4 (\( |dz| = 0.17\) , roughly twice the others) and misses C1/C2/C3 (\( dz \approx 0.08–0.10\) ). At QASPER’s n = 4,303 the larger sample overwhelms the inflated errors, and it confirms all three non-null contrasts.

The disagreements between the bootstrap and the MixedLM on the small Wikipedia-FA effects are therefore a specification artifact of the secondary model, not evidence that the effects are fragile. The bootstrap is correctly specified and primary. The paper reports the MixedLM for pre-registration fidelity only.

The classifier assigns each accepted document to one of six keyword-defined topic buckets. It matches the document’s non-hidden Wikipedia categories against a fixed keyword table. The first matching bucket in table order wins. An other bucket is the fallback:

The "balancing" is a single hard ceiling: max_per_bucket = 60. The seeded walk enforces it. It rejects a candidate whose bucket already holds 60 accepted documents. There is no minimum-per-bucket guarantee and no round-robin. The cap only limits over-represented topics. It does not equalize them.

The cap binds on the two largest buckets. Both land exactly at 60. Without it, history_war and other would each be larger. 60% of the corpus still sits in two buckets. This reflects the genuine topic skew of Featured Articles. The paper discloses this skew rather than hides it, and the skew strengthens the generalization claim. The replicated organization effects hold over a topic-imbalanced corpus, not a curated one.

Gold structure is the article’s own heading tree. Two layers make its character offsets trustworthy.

Exact by construction. The pipeline builds the flat document string paragraph by paragraph. It records each paragraph’s as it emits the paragraph. It reads gold section offsets directly from this paragraph table. It never recovers them by searching for a heading’s text. A text search is the step that would introduce off-by-N drift.

Mechanically verified. Segment verification returns a list of violations. An empty list means verified. It enforces that the sections:

• tile the document with no gaps or overlaps. Each segment starts exactly where the previous one ended. The final segment ends at the last paragraph.
• have offsets that agree with the paragraph table exactly. Otherwise it reports "char offsets disagree with paragraph table".
• have in-bounds paragraph ranges and legal nesting levels (\( 1 \leq level \leq MAX\_LEVEL=4\) ).

Relevance runs purely as character-span overlap (C). These offsets are therefore load-bearing. An off-by-N error would silently corrupt every relevance judgment. The same offset discipline applies to induced structure (H). Induced structure emits paragraph index ranges rather than raw offsets. This makes "structure without content modification" a verified property and holds the gold and induced arms to one integrity standard.

Placebo prefixes are real heading paths from other documents. The pipeline buckets unique (document, path) pairs by depth. It permutes them within depth-bucket toward a cross-document derangement under a fixed seed. It counts and reports residual same-document assignments. The result matches real breadcrumbs in format, depth, and length distribution. It carries no information about the chunk it attaches to. This is a semantically null treatment in the strict sense.