RAG Chunking Strategies That Actually Improve Recall

Verdict: What Actually Works

Start with a fixed-size splitter at a few hundred tokens, test it on your own queries before you try anything exotic, keep overlap small, and fix context loss with contextual retrieval instead of bigger chunks. That default holds up across the benchmarks and production studies cited in this article, and it costs almost nothing to try. Four decisions carry most of the result: the splitter you pick, the chunk size you start with, the overlap you allow, and the way you restore document context after the split.

Chunking looks like a formatting chore. It behaves like a retrieval decision. Every chunk becomes one vector, and the retriever can only return what the chunker preserved. Get the split wrong and the right document sits in your index while the retriever returns a neighbor that says almost nothing. This article compares six strategies, gives you a starting grid of sizes, and shows you a small eval that tells you which one wins on your corpus.

Chunking Decides What Your RAG App Can Find

An embedding model compresses a whole chunk into a single vector. A chunk that covers three topics at once produces a vector that points at none of them with confidence. A chunk cut too small keeps a fact but drops the sentence that explains what the fact refers to. Both failure modes look identical from the outside: the user asks a fair question and gets a confident answer built on the wrong passage.

Also Read:

LoRA Fine-Tuning Explained: Adapt LLMs for Less

Two more constraints shape the choice. First, the generator has a context window ceiling, so retrieved chunks compete for space with the prompt, the conversation history, and the answer itself. Stuffing the window with long chunks leaves less room for reasoning and raises cost per answer. Second, models show a lost-in-the-middle effect: passages buried in the middle of a long context get less attention than passages at the start or the end. Long chunks make that effect worse because each retrieved unit carries more filler around the one relevant sentence.

Read how embeddings turn chunks into searchable vectors before you tune anything else, and pair the choice with choosing a vector database for your chunks. The store matters less than the split: a strong index cannot rescue chunks that blurred their own meaning at ingest time.

Six Chunking Strategies Compared

Fixed-size token splitting cuts text every N tokens with no regard for sentence boundaries. Recursive character splitting tries a list of separators in order, paragraph breaks first, then sentences, then words, so chunks stay closer to natural units. Sentence and paragraph splitting respects one boundary type only. Structure-aware splitting follows the document format: headings in Markdown, tags in HTML, sections in PDF, functions in code. Semantic splitting measures embedding distance between sentences and cuts where the topic shifts. LLM and contextual chunking asks a model to write a short context for each chunk or to decide the boundaries itself.

Also Read:

LLM Quantization Explained: Fit Bigger Models in Less VRAM

Cost separates the table into two groups. The first four strategies run as string operations, so you can re-chunk a whole corpus in minutes. Semantic splitting adds one embedding pass over sentence pairs. LLM chunking adds a model call per chunk, which is fine for a thousand documents and painful for a million. Start cheap, measure, and spend on model calls only where the eval shows context loss.

Chunk Size and Overlap: The Numbers to Start With

Test a grid, not a single size. Pinecone’s chunking guide recommends fixed-size chunking as the starting point for most teams and suggests exploring smaller sizes such as 128 or 256 tokens alongside larger sizes such as 512 or 1024 tokens. The guide frames the grid as the method: run the sizes, score retrieval, and keep the winner. Guessing one size from a blog post and shipping it skips the only step that adapts the choice to your corpus.

Answer shape moves the optimum. A 2025 pre-print from Bhat and co-authors found smaller chunks of 64 to 128 tokens optimal for datasets with concise, fact-based answers, while larger chunks of 512 to 1024 tokens improved retrieval for datasets that need broader contextual understanding. A support bot answering “what is the refund window” wants the small end. A research assistant summarizing a policy debate wants the large end. One corpus can hold both query types, which is why a single global size often disappoints.

Also Read:

AI Agents vs Workflows: When to Use Which

Chroma’s July 2024 technical report measured the same lesson with numbers. On a 474-query dataset with text-embedding-3-large and five retrieved chunks, recursive splitting at 800 tokens with 400 overlap scored 85.4 recall, while recursive splitting at 400 tokens with zero overlap scored 89.5. An LLM chunker built on GPT-4o scored 91.9 and a cluster-based chunker scored 91.3, while the default semantic chunker scored 83.6. The widely shipped 800-token default with heavy overlap landed below average in that study, and the report put the total effect of chunking strategy choice at up to 9 percent of recall. Treat those figures as one study’s results on one dataset, not as universal constants.

Overlap deserves skepticism. A 2026 industry pre-print from Bennani and Moslonka reported that overlap provided no measurable benefit and raised indexing cost, that sentence chunking was the most cost-effective method and matched semantic chunking up to about 5,000 tokens, and that a context cliff reduced quality beyond about 2,500 tokens of retrieved context. Label that study for what it is, a short pre-print, but its direction matches the Chroma result: the 400-token overlap in the default configuration bought little and cost a lot. Start with zero or small overlap, add overlap only if your eval shows boundary misses, and keep an eye on the total context you feed the generator.

External sources for this section: Pinecone’s chunking strategies guide, Chroma’s evaluating chunking report, the 2025 chunk-size pre-print, and the 2026 industry pre-print on overlap and context ceilings.

Also Read:

Prompt Injection Explained: How It Works and How to Stop It

Contextual Retrieval: Fixing What Splitting Breaks

Splitting creates a specific injury: a chunk that made sense inside its document loses the document context once it stands alone. A paragraph about “the policy” means nothing if the retriever cannot see which policy the document discusses. Anthropic’s contextual retrieval answers that injury by prepending a short context, usually 50 to 100 tokens, to each chunk before embedding it and before building the BM25 index. The context names the document and the section the chunk came from, so the chunk carries its own provenance.

The measured effect is large. Anthropic reported that contextual embeddings reduced the top-20-chunk retrieval failure rate from 5.7 percent to 3.7 percent, that combining contextual embeddings with contextual BM25 cut the rate to 2.9 percent, and that adding reranking cut it to 1.9 percent. In relative terms, the method reduced failed retrievals by 49 percent, and by 67 percent with reranking. Those numbers come from Anthropic’s September 2024 writeup, so read them as the vendor’s benchmark result, not as a promise for your corpus.

Cost follows the same honest framing. In Anthropic’s worked example from 2024, generating contextualized chunks for the example corpus cost about one dollar one time, based on the pricing and model of that date. Pricing has moved since then, so re-price the step before you quote it in a budget. The structural point survives the price change: contextual retrieval charges once at index time and pays on every query after that, while bigger chunks charge on every query forever.

Cheaper fixes exist when model calls per chunk feel too rich. Parent-document expansion retrieves a small chunk, then hands the generator the larger section around it. Neighbor expansion pulls the chunks before and after a hit at query time. Both restore context without touching the index, and both pair well with prompt caching to keep corpus-in-prompt costs down. Anthropic’s same writeup offered a rule of thumb worth repeating with its date attached: when a knowledge base fits under about 200,000 tokens, feeding the whole base in the prompt can beat retrieval outright, with latency and cost advantages from caching. Context windows and cache pricing have moved since 2024, so treat the threshold as a prompt to test, not as a current constant.

Once retrieval runs in production, watch what it returns. Tracing what your production retrieval app actually retrieved shows you the misses your eval never saw, and those traces tell you whether a chunking change helped or hurt after shipping.

How to Evaluate Chunking on Your Own Data

Build a 20-query mini-harness before you commit to a configuration. Collect twenty questions your users ask, with the passage each question should retrieve. Build two or three splitter variants, embed each variant into a separate index, and score recall at k for each variant against the same twenty questions. Include generation cost in the verdict, because a configuration that retrieves a fraction better at triple the token cost loses on the invoice.

Four steps keep the harness honest. First, freeze the question set before you look at results, or you will tune toward the questions you like. Second, change one variable at a time: size, then overlap, then splitter, never all three at once. Third, score retrieval failure separately from answer quality, because a good generator can mask a bad retriever for a while. Fourth, rerun the harness after any corpus change that alters document structure, such as a new template or a new document type.

One more trap: do not A/B chunkers on a stale embedding model. OpenAI’s announcement for its newer embedding models reported MIRACL scores rising from 31.4 percent to 54.9 percent and MTEB scores rising from 61.0 percent to 64.6 percent when moving from text-embedding-ada-002 to text-embedding-3-large. A chunking comparison run on an old model measures the model gap, not the chunking gap. Upgrade the embedding model first, then compare splitters. Building an eval harness that catches real failures covers the harness design in more depth, and OpenAI’s embedding model announcement carries the score details.

Conclusion: Ship the Default, Then Measure

Ship this configuration this week: recursive splitting at 400 tokens with zero overlap for prose corpora, structure-aware splitting for docs and code, and contextual retrieval or neighbor expansion wherever chunks lose document context. Run the 20-query harness against it, and keep the configuration only if it wins on your questions. Revisit the choice when two conditions change: your query shape shifts toward fact lookup or toward broad synthesis, or your corpus structure changes with a new document type.

Chunking rewards measurement over debate. The studies agree on direction and disagree on exact numbers, because corpora differ and query shapes differ. Your corpus gets a vote, and the harness counts it. Run the eval, keep the winner, and read when retrieval is the right call vs fine-tuning when you wonder whether retrieval is even the right tool for the job. Start with the default above, measure it on your own data, and let the numbers pick your chunk size.