Chunking for RAG: the evidence on size, overlap and structure

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A chunk is the piece of a document stored as one searchable item, and often the text the generator later reads. Size, overlap and structure are separate choices. The experiments do not meet on one window length. They do show when a short block, a heading, or a rewritten fact beats a long span.

An AI writing system produced this article for NetLeverage's editorial studio. Figures are those printed in the sources. Nothing was run in a lab, and no NetLeverage product behaviour is claimed.

Why it matters

Lewis et al. condition a generator on whatever the index returns, and the index returns the units you cut. A dense retriever maps each unit to one vector, so a longer unit packs more facts into that vector. A cut can also drop the name a later sentence depends on: in Günther et al., "its" and "the city" refer to Berlin, which appears only in the first sentence. Extra overlap and extra structure change storage and the prompt. Yepes et al. raised a retrieval score by merging chunk indexes, then could not generate because the combined text no longer fitted.

Three lengths

LlamaIndex's node-parser page describes a sentence-window parser this way: each node is one sentence, the surrounding sentences sit in metadata, and that metadata stays hidden from the embedding model and the language model until a later step replaces the text. The page's default-settings example uses three sentences on either side.

Haystack's DocumentSplitter page, labelled version 3.4 on 10 November 2026, splits by word, sentence, passage, page, line, period, token, or a function. It says the Chroma and Pinecone stores drop overlapping information, so the shared span can sit in the vectors without a record of the overlap.

The unit the original systems indexed

Lewis et al. index the December 2018 English Wikipedia dump as disjoint 100-word chunks (21 million documents in their count). The retriever is Karpukhin et al.'s dense passage retriever and the generator is BART.

Karpukhin et al. use the dump of 20 December 2018, strip tables, infoboxes, lists and disambiguation pages with the DrQA cleaner, and cut 21,015,324 disjoint 100-word passages. Each passage is prefixed with the article title and a separator, so a block that never repeats the subject still carries the page name. On that prose, natural paragraphs were worse for retrieval and for answer accuracy than the fixed blocks, and overlapping passages were not advantageous. Both results are about Wikipedia after the structured elements were removed.

Their metric is the share of questions whose answer string is inside the top 20 or top 100 passages. On Natural Questions the single-dataset dense retriever reaches 78.4% top-20 accuracy, against 59.1% for Lucene BM25 with the $b=0.4$, $k_1=0.9$ settings they tuned. The passage in that comparison is the 100-word block plus the title.

Characters, tokens and words

A setting called "512" is not one setting. Wang et al. count tokens and hold overlap at 20 tokens. Yepes et al. use token windows of 128, 256 and 512, plus a 2,048-character merge rule. Merola and Singh fix one baseline at 512 characters. Günther et al. use 256 tokens in the main late-chunking table.

Library defaults opened the same day use other counters.

langchain-text-splitters 1.1.3 (tag langchain-text-splitters==1.1.3, PyPI upload 2 October 2026) defaults TextSplitter to chunk_size=4000 and chunk_overlap=200, measured by Python len (characters) unless another counter is passed. from_huggingface_tokenizer counts Hugging Face tokens. from_tiktoken_encoder uses tiktoken, and the helper's default encoding name is gpt2 when no model is passed. RecursiveCharacterTextSplitter tries a blank line, a newline, a space, then each character. Overlap greater than the size raises.

Unstructured's chunking page (no package version printed) defaults max_characters to 500 and overlap to 0. Overlap applies when an oversized element is split. Overlap on ordinary chunks needs overlap_all=True, which the page says can pollute a clean element boundary. A table stays apart from other elements. by_title closes the chunk when a title starts.

LlamaIndex's modules page shows examples, not labelled defaults: sentence and token splitters at 1024 and overlap 20, a hierarchy of 2048, 512 and 128, and a semantic breakpoint percentile of 95.

For a fixed window the step is the window minus the overlap, so a shorter step stores more vectors. That geometry is a scale for the tables, not a paper formula. A separator-based splitter stops early when a paragraph fits inside the budget.

What the size comparisons measured

One filing, about 170 generated questions

Wang et al. embed with text-embedding-ada-002 and score with LlamaIndex's evaluation module. Generation in that section is Zephyr-7B-alpha; GPT-3.5-turbo is the judge. The corpus is the first 60 pages of a Lyft 2021 filing. About 170 queries are written by a model. Overlap stays at 20 tokens.

On this set 256 and 512 are close, and 2048 drops faithfulness from 97.59 to 80.37. With LLM-Embedder, a 175-token block inside a 512-token parent (overlap 20) moves faithfulness from 95.74 to 96.67 (small-to-big) or 97.41 (sliding window). Those gains are small beside the long-window drop. The queries were generated from one filing.

One hundred questions

Ammar et al. (arXiv:2505.08445) vary store, chunk policy, reranking and temperature on 100 questions. Embeddings are text-embedding-3-small (1,536 dimensions). Generation is gpt-4o-mini-2024-07-18. Selected rows from Table IV:

On these rows, 1024 tokens with overlap 128 is the strongest naive setting and the fastest. Overlap 512 on that window lowers all three quality scores. Every semantic-breakpoint row sits under the weakest naive faithfulness shown (0.887). These are selected rows: at 4096 tokens, overlap 512 beat overlap 128 on precision and recall (0.851/0.830 versus 0.838/0.827). The authors limit this to the English 100-question set and these models. A later corrective-retrieval rerun in the same paper reports 99% retrieval relevance for overlap 512 with reranking and Chroma, so the overlap result is not one direction even here.

Overlap

Günther et al. add a 16-token overlap to a 256-token window and compare it with none, using jina-embeddings-v2-small. nDCG@10, in percent:

They conclude the scores are similar. The main text says the section 4.1 runs were non-overlapping, while the naive "overlap" column matches Table 2's naive fixed-size row. Use the appendix labels. That mismatch was still in the HTML on 10 November 2026.

Our interpretation is that small overlap can repair boundaries while large overlap can add near-duplicate vectors. Karpukhin and Günther compared overlap with no overlap and reported no clear overall gain, although Günther's naive results improved on three of four sets. Ammar compared 128 with 512 tokens of overlap and included no zero-overlap row; that study cannot establish whether enabling overlap helps.

Semantic breakpoints

Qu, Bao and Tu split with spaCy en_core_web_sm, then either group a fixed number of sentences (optionally sharing one), cut where neighbour distance exceeds a threshold, or cluster sentences that need not be adjacent. Clustering uses

with $n$ the sentence count. The paper says $\lambda =1$ mirrors the fixed-size chunker. Negative cosine is treated as zero. Scores below are F1@5 for dunzhang/stella_en_1.5B_v5 at each chunker's best setting.

Breakpoint chunking leads on stitched sets of about 100 sentences: Natural Questions 63.93 versus 43.79, MIRACL 81.89 versus 69.45. On original long documents the fixed-size chunker leads or ties: HotpotQA 90.59 versus 87.37, MS MARCO 93.58 versus 92.23, ConditionalQA 68.11 versus 64.44. Evidence-retrieval gaps are tenths of a point (ExpertQA 47.11 versus 47.08, TechQA 28.98 versus 28.49, Qasper 8.66 versus 8.16). With gpt-4o-mini and five chunks, BERTScore matches to two decimals on three of five sets. The authors say the cost is not justified as a default. The sentence vectors were context-free, and they had no human chunk labels. The gains show up when a file is several documents glued together.

Putting the missing name back

Chen et al. (EMNLP 2024) compare 100-word passages, sentences, and propositions: one fact each, minimal, and readable alone. FactoidWiki is the 13 October 2021 English Wikipedia dump. GPT-4 writes seed lists; Flan-T5-large is fine-tuned to emit them. On 50 passages, most propositions were supported by the source, and a smaller share could not stand alone.

Average Recall@20 rises by 10.1 points for their unsupervised dense retrievers and by 2.7 for the supervised ones. On Natural Questions, TriviaQA and WebQuestions the supervised retrievers are described as comparable across granularities. DPR itself was slightly worse at proposition and sentence level than at passage level on three of the four sets it was trained on. With LLaMA-2-7B and a 500-token cap, propositions raise exact match by 4.1, 3.2, 2.7 and 2.8 for SimCSE, Contriever, DPR and GTR against passages. On narrative books, Duarte et al. get the opposite: proposition DCG@20 46.19 and Recall@20 55.54, against 54.72 and 74.35 for recursive splitting.

Günther et al. run the transformer on the long text first, then mean-pool a span whose token states have already attended to the document:

The chunker still supplies the boundaries. Similarity of the query "Berlin" to "Its more than 3.85 million inhabitants…" rises from 0.7084 to 0.8249 with jina-embeddings-v2-small-en. The sentence that already names Berlin barely moves (0.8486 to 0.8495). On four smaller BEIR sets, average nDCG@10 rises from 52.2 to 54.0 (256-token windows), 52.4 to 54.3 (five-sentence windows) and 52.4 to 53.8 (semantic sentences), across three embedders: their relative gain of about 2.7% to 3.6%. Past the model window, macro-chunks overlap and the extra token states are dropped so each token is pooled once. The method loses when surrounding text is irrelevant (their needle and passkey sets) and on some reading sets once chunks are large.

Merola and Singh, using 512-character segments, do not reproduce a general gain. Stella-V5 on NFCorpus is 0.443 early and 0.445 late at nDCG@5. BGE-M3 falls from 0.246 to 0.070. Stella-V5 on MS MARCO falls from 0.630 to 0.503. They mark the paper as not yet peer reviewed. Their contextual-prefix comparison used Phi-3.5-mini-instruct, BM25 weight 0.25 against dense weight 1, and a reranker, on about 50 queries, because of computing cost and hardware constraints including a 24 GB GPU. Jina-V3 on NFCorpus showed a small contrary gain, 0.374 early versus 0.380 late at nDCG@5.

Finardi et al. test LlamaIndex's sentence window on a Brazilian Portuguese Harry Potter novel, about 140,000 cl100k_base tokens, cut into 140 chunks of 1,000 tokens with no overlap. GPT-4 wrote each question so the answer sits inside one chunk. The window scores 5.77 for gpt-4, a 23.5% drop against their relative maximum: repeated names make a bare sentence ambiguous, and no answer straddles a cut. Perfect recall at nine chunks lost to three chunks after a reranker.

Structure the document already has

Yepes et al. keep headings and tables intact on FinanceBench: 80 documents, 141 questions (some of the 150 questions had missing files). Chipper labels elements. A new chunk starts at a title or a table, the table stays whole, and other elements merge under 2,048 characters. Keywords or a summary from GPT-4, or a two-sentence prefix, can be attached.

Token indexes hold 64,058, 32,051 and 16,046 chunks at 128, 256 and 512 tokens. The element index holds 20,843. Page accuracy is 73.05% (256 tokens) and 68.09% (512), against 67.38% for the prefix-and-table index. BLEU against the labelled evidence is 0.231, 0.250 and 0.400 in that same order. Manual answer accuracy is 35.46%, 36.88% and 48.23% at 128, 256 and 512, and 53.19% for the keyword index and the prefix-and-table index. A GPT-4 judge scored that prefix index at 41.13%. Merging token indexes reaches 83.69% page accuracy at 112,155 chunks; merging element variants reaches 84.40%, ROUGE 0.568 and BLEU 0.452, at 62,529 chunks, and the generator could not read them. Element chunks won answer accuracy and lost page accuracy to the 256-token window. Unstructured's by_title page says a short list item can be labelled a title, and combine_text_under_n_chars can merge the tiny sections that follow.

Sarthi et al. (RAPTOR) build a tree. Leaves are 100 tokens; a sentence that would cross the limit moves wholly to the next chunk. SBERT (multi-qa-mpnet-base-cos-v1) embeds them, a cluster is summarised with gpt-3.5-turbo, and the summary is clustered again. On 20 QASPER stories, flattening the tree and stopping at a token budget beat a fixed count per layer. 2,000 tokens (about 20 nodes) was best on that slice.

With UnifiedQA-3B fixed, QuALITY accuracy moves from 54.9% to 56.6% (SBERT), 49.9% to 52.1% (BM25) and 53.1% to 54.7% (DPR). QASPER F1 moves from 36.23 to 36.70, 26.47 to 27.00 and 31.70 to 32.23. Reported RAPTOR F1 with GPT-3, GPT-4 and UnifiedQA is 53.1, 55.7 and 36.6, ahead of DPR by 1.8, 2.7 and 4.5. The abstract's large QuALITY jump is RAPTOR plus GPT-4 at 82.6% against a previous best of 62.3%, which changes the reader as well as the index.

Duarte et al. ask Gemini 1.0 Pro which paragraph starts a new topic. Paragraphs accumulate to a budget $\theta$; a sweep from 450 to 1,000 tokens picks 550. GutenQA is 100 public-domain books and 3,000 needle questions, embedded with text-embedding-ada-002.

The abstract's "7.37%" is that 7.37-point DCG@20 gap. Recall@20 moves from 74.35 to 77.92. It costs more than recursive splitting and uses a closed model. On 280 autobiography questions it beat most automatic baselines, including recursive splitting, and fell short of manual chunks.

What to choose and why

Count in the embedder's units. Four thousand characters, 512 tokens, and 500 characters are three different windows.

Risks and limits

• Questions written after the cut, or generated from the corpus, flatter that cut. Finardi et al., Wang et al. and Duarte et al. each did one of those. Mark evidence on the original pages.
• Breakpoint chunking won on stitched documents and lost or tied on the original long documents in Qu et al.
• Late chunking raised Günther et al.'s BEIR average and lowered BGE-M3 on NFCorpus and Stella-V5 on MS MARCO in Merola and Singh.
• Yepes et al. depend on Chipper's labels. The Unstructured page says a false title slices too finely. Their GPT-4 judge scored 41.13% where people scored 53.19% on the same prefix index.
• Nine chunks lost to three after reranking in Finardi et al. Merged indexes overflowed the generator in Yepes et al.
• The 100-word result is Wikipedia prose after tables and lists were removed. LangChain numbers are the 1.1.3 tag; LlamaIndex numbers are examples; the Unstructured page gave defaults and no release version.
• No NetLeverage product behaviour is claimed. These steps were not executed.

Sources

Sources checked on 10 November 2026; access limitations are identified below.

• Lewis et al., arXiv:2005.11401 — https://ar5iv.labs.arxiv.org/html/2005.11401
• Karpukhin et al., arXiv:2004.04906 — https://ar5iv.labs.arxiv.org/html/2004.04906
• Wang et al., arXiv:2407.01219 — https://ar5iv.labs.arxiv.org/html/2407.01219
• Ammar et al., arXiv:2505.08445 — https://arxiv.org/html/2505.08445
• Qu, Bao and Tu, arXiv:2410.13070 — https://arxiv.org/abs/2410.13070
• Günther et al., arXiv:2409.04701v3 — https://arxiv.org/html/2409.04701v3
• Merola and Singh, arXiv:2504.19754 — https://arxiv.org/abs/2504.19754
• Chen et al., arXiv:2312.06648 — https://ar5iv.labs.arxiv.org/html/2312.06648
• Duarte et al., arXiv:2406.17526 — https://ar5iv.labs.arxiv.org/html/2406.17526
• Yepes et al., arXiv:2402.05131 — https://ar5iv.labs.arxiv.org/html/2402.05131
• Sarthi et al., arXiv:2401.18059 — https://ar5iv.labs.arxiv.org/html/2401.18059
• Finardi et al., arXiv:2401.07883 — https://ar5iv.labs.arxiv.org/html/2401.07883
• langchain-text-splitters 1.1.3 — https://github.com/langchain-ai/langchain/blob/langchain-text-splitters==1.1.3/libs/text-splitters/langchain_text_splitters/base.py
• LlamaIndex node parsers — https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/modules/
• Haystack DocumentSplitter, docs labelled 3.4 — https://docs.haystack.deepset.ai/docs/documentsplitter
• Unstructured chunking — https://docs.unstructured.io/open-source/core-functionality/chunking

This article was originally published on 10 November 2026.