Chunking Strategies for RAG: How to Split Documents So Retrieval Actually Works
By Abhishek Patoliya · 2026-09-30
When a retrieval-augmented generation (RAG) system gives a bad answer, the instinct is to blame the model or rewrite the prompt. More often the real problem happened earlier, in a step that gets a single line in most tutorials: chunking, the decision about how to cut documents into pieces before they are embedded and stored.
Cut too small and a passage loses the context that made it meaningful. Cut too large and retrieval becomes vague, you waste space in the prompt, and the useful sentence gets buried. Cut in the wrong place, in the middle of a table or between a heading and its content, and the right information may exist in your index but never be retrievable as a coherent unit.
This article explains what chunking is and why it matters, walks through the main strategies with their trade-offs, covers the awkward material (tables, code, PDFs, transcripts), and shows how to test chunking decisions with evidence instead of folklore. It assumes you know the basics of RAG. If you do not, start with RAG explained and the RAG integration architecture guide.
Why chunking exists at all
RAG, as introduced in the 2020 paper by Lewis and colleagues, combines a model's built-in knowledge with a searchable index of documents: the system retrieves relevant passages and conditions its answer on them. n8n's glossary describes RAG as a technique for giving language models access to new information from external sources to improve responses.
Two practical constraints force you to break documents up.
Embedding models work on limited spans of text. An embedding turns a piece of text into a vector, a list of numbers that represents its meaning; n8n's glossary describes embeddings as numerical representations of data using vectors. Each embedding model has a maximum input length, so a long document cannot be embedded in one piece, and check your model's documented limit rather than assuming it. Even where a whole document fits, a single vector for a 40-page report averages many topics into a blur that matches nothing well.
The model's context window is finite and costly. Whatever you retrieve goes into the prompt, where you pay for every token and where irrelevant text can distract. You want to retrieve the specific passage that answers the question, not the whole handbook.
So you split documents into chunks, embed each, and store them in a vector store. LangChain's RAG tutorial describes exactly this indexing pipeline: load documents, split them, embed the pieces and store them. In its example, 14 documentation pages became 782 chunks, using a recursive character splitter with chunks of 1,000 characters and 200 characters of overlap. That is a reasonable starting point, and it is worth being clear about what it is: a default, not a rule. The tutorial itself notes that selecting only the passages relevant to a question is a non-trivial task.
At query time, the question is embedded, the nearest chunks are retrieved, and they are placed in the prompt for the model to answer from. Everything downstream depends on whether the right chunk exists and can be found.
The core trade-off
Every chunking decision balances two opposing forces.
Precision favours small chunks. A small chunk is about one thing, so its embedding represents that thing sharply, and a query about that thing matches it strongly. Small chunks also keep the prompt lean.
Context favours large chunks. A larger chunk carries the surrounding information that makes a statement interpretable: which product, which region, which version, what "it" refers to. Without that, a retrieved sentence may be technically relevant and practically useless, or worse, misleading.
Consider a policy document containing the sentence "Refunds are available within 30 days." If the chunk boundary falls so that the heading "Digital products" sits in the previous chunk, the retrieved sentence loses its scope. The model may confidently tell a customer about a physical-goods policy that does not apply.
Most of chunking strategy is about getting both: small enough to be precise, with enough context to be correct.
Strategy 1: fixed-size chunks
The simplest approach: cut every N characters or tokens, optionally with overlap.
How it works. Choose a size (say 800 tokens) and an overlap (say 100). Slide along the text, emitting chunks.
Strengths. Trivial to implement, predictable in size, works on any text, easy to reason about cost.
Weaknesses. It ignores meaning entirely. It will cut mid-sentence, split a list from its heading, separate a question from its answer, and slice tables in half. Overlap helps a little by repeating boundary text, but does not restore structure.
When to use it. Prototypes, homogeneous prose (like plain articles), and as a baseline against which to measure smarter methods.
Strategy 2: recursive splitting
A refinement used by many frameworks. The splitter tries a hierarchy of separators, typically paragraph breaks first, then line breaks, then sentences, then words, and only falls back to a finer separator when a piece is still too large.
How it works. Split on double newlines; if any resulting piece exceeds the size limit, split that piece on single newlines; then on sentence boundaries; and so on. Small adjacent pieces are merged back up to the target size.
Strengths. Respects natural boundaries where it can, so chunks are more likely to be coherent paragraphs. It is the reason this is the common default: it is nearly as easy as fixed-size splitting and produces noticeably cleaner chunks. It is the approach in the LangChain tutorial's example.
Weaknesses. It still does not know about headings, tables or semantic topic changes. A long paragraph may still be cut awkwardly, and a short heading may be merged with the wrong neighbour.
When to use it. A good first choice for prose-heavy content, such as articles, help pages and policies, especially when you have not yet measured anything.
Strategy 3: structure-aware splitting
If your documents have structure, use it. Split on headings, sections, list items, table rows, slide boundaries, FAQ entries or code functions, rather than on character counts.
How it works. Parse the document into its logical units (HTML headings, Markdown sections, Word styles, PDF layout blocks), and make each section a chunk, splitting further only when a section is too large. Carry the heading path with each chunk: "Returns policy > Digital products > Refund window."
Strengths. Chunks correspond to the units people actually think in, which is also how questions tend to be framed. Heading paths supply context cheaply. FAQ documents become ideal: each question-and-answer pair is a natural chunk.
Weaknesses. It needs reliable parsing, and real documents are messy: PDFs converted to text may lose headings entirely, and inconsistent formatting breaks the rules. Sections vary wildly in length, so you still need a fallback for oversized ones and a merge rule for tiny ones.
When to use it. Whenever the source has usable structure, which for most business documentation, wikis and knowledge bases it does. In practice it often gives the largest improvement for the least conceptual complexity.
Strategy 4: semantic chunking
Instead of using formatting, split where the topic changes. A common method embeds each sentence (or small group), measures the similarity between adjacent ones, and starts a new chunk where similarity drops sharply.
Strengths. Can produce topically coherent chunks even in unstructured text such as transcripts or long unformatted documents.
Weaknesses. More complex and more expensive, since it embeds at a finer grain during indexing. Results depend on thresholds you must tune, chunk sizes become irregular, and the gains over good structure-aware splitting are not guaranteed. Evaluate it against simpler methods on your data before adopting it.
When to use it. Unstructured, long-form material where structure-based splitting has nothing to hold on to, and where you can afford the experimentation.
Strategy 5: sentence windows and small-to-big retrieval
This pattern separates what you match on from what you give the model.
How it works. Embed small units (a sentence, or a short paragraph) for sharp matching. Store, for each, a pointer to a larger context: the surrounding paragraphs, or the whole section. At query time, match on the small unit, but return the larger parent to the model.
This is often called parent-child or small-to-big retrieval. Precision comes from the small embedded unit; context comes from the larger returned unit.
Strengths. Addresses the precision-versus-context tension directly. Particularly good when answers depend on neighbouring text.
Weaknesses. More moving parts: you need to store the relationships, handle the case where several small hits share the same parent (deduplicate), and control how much parent text goes into the prompt. It uses more tokens per retrieval than returning small chunks.
When to use it. When you observe that retrieval finds the right spot but the retrieved text lacks enough context for a correct answer.
Strategy 6: adding context to each chunk
Regardless of how you split, you can improve retrieval by attaching context to every chunk before embedding.
Prepend a header. Include the document title, section path and possibly a date or version: "Company handbook, Returns policy, Digital products (updated 2026-03)." This ensures the embedding reflects the scope, and the model sees it when the chunk is retrieved.
Store metadata separately. Keep fields such as source, section, author, document type, language, effective date, product, region and access permissions alongside the vector. Metadata enables filtering ("only documents for the UK") that pure similarity cannot.
Consider a generated summary or context line. Some teams have a model write a one-sentence description of where each chunk sits in its document and embed that with the chunk. It can help with ambiguous fragments, but it adds cost and another place for errors, and you should test that it improves retrieval before adopting it.
Choosing sizes: how to think, not what to memorise
There is no universally correct chunk size, and you should distrust anyone who states one without knowing your documents. Factors to consider:
The shape of your questions. Short factual questions ("what is the return window?") suit small chunks. Explanatory questions ("how does the approval process work?") suit larger ones.
The shape of your documents. Dense reference material with self-contained entries tolerates small chunks. Narrative or argumentative text needs more context.
The embedding model's limit and behaviour. Do not exceed the documented maximum input length. Beyond that, models can differ in how well they represent long spans, so test rather than assume.
How many chunks you will retrieve. If you retrieve five chunks, their combined size must fit comfortably in the prompt along with instructions and the question. Smaller chunks let you retrieve more of them; larger chunks let you retrieve fewer.
Cost. More, smaller chunks mean more embeddings to compute and store at indexing time, and possibly more tokens to send if you retrieve many. Indexing cost is usually a one-time expense per document version; prompt cost recurs on every query.
A sensible way to start: use recursive or structure-aware splitting with a chunk size in the neighbourhood of a few hundred tokens (LangChain's example of 1,000 characters is roughly in that range for English prose), a modest overlap of around 10 to 20 percent, and then measure. The next sections explain how.
Overlap: useful, but not a fix
Overlap repeats a slice of text at the end of one chunk and the start of the next. Its purpose is to avoid losing information that straddles a boundary: if a key sentence ends up split, at least one chunk contains most of it.
Benefits are real but modest. Costs are also real: more storage, more embeddings, and duplicated content that can appear twice in retrieved results, wasting prompt space. Heavy overlap (say 50 percent) usually indicates that the underlying split points are poor; it is better to fix the boundaries than to paper over them.
If you use overlap, deduplicate at retrieval time so the same text is not passed twice, and treat the overlap size as another parameter to test.
The awkward material
Real corpora are not tidy prose. Each of these deserves specific handling.
Tables
A table sliced into arbitrary chunks is often unintelligible: rows without headers, columns without labels. Better options:
Keep small tables whole as a single chunk, including the caption and header row.
For large tables, chunk by row groups but repeat the header row and caption in each chunk.
Convert rows to sentences ("Product X, region UK, price 49") if the table is a lookup, so that each row is self-describing.
If the questions are really database queries ("what was revenue in Q3?"), consider querying the data directly instead of embedding it. See RAG for structured data.
Code
Split by function, class or logical block, not by character count, and include the file path and signature as context. A function cut in half is close to worthless.
PDFs and scanned documents
PDF text extraction can scramble reading order, merge columns, drop headings and strip formatting. Chunking on top of bad extraction cannot recover the lost structure. Inspect the extracted text yourself for a sample of documents before you build anything on it. For scanned documents, the quality of the OCR step limits everything downstream. Intelligent document processing covers extraction in more depth.
Transcripts and chat logs
Speaker turns are the natural unit, but a single turn may be too short to stand alone. Group consecutive turns into windows, retain speaker labels and timestamps, and consider adding a topic or meeting title for context.
Slides
A slide is often a fragment ("Q3 priorities: reduce churn") whose meaning depends on the speaker's notes. Include the slide title, deck title and any notes in the chunk.
Duplicate and near-duplicate content
Templates, boilerplate footers, repeated disclaimers and multiple versions of the same document create clutter in results. Remove boilerplate before indexing, and deduplicate near-identical chunks, or the retriever may return five copies of the same paragraph.
Multiple versions and dates
If old and new policies are both indexed, the retriever cannot tell which is current unless you tell it. Store effective dates and status in metadata, filter to current versions by default, and keep superseded ones out of the index unless there is a reason.
Metadata and permissions
Chunking is also where you decide what information travels with each piece, and permissions are the item most often forgotten.
If your index mixes documents with different audiences, every chunk must carry its access information, and retrieval must filter on it before passing anything to the model. Otherwise a user's question could pull in text they are not entitled to see. OWASP includes vector and embedding weaknesses in its 2025 Top 10 for LLM Applications, and permission-aware retrieval is a central concern; see permission-aware retrieval for RAG.
Also remember that retrieved text is untrusted input to the model. The LangChain tutorial warns that retrieved content can carry indirect prompt injection, meaning instructions embedded in a document that try to override the system prompt, and notes that mitigations such as labelling sources and treating retrieved content as data do not provide reliable protection on their own. Chunking does not fix this, but you should design retrieval, prompts and downstream permissions with it in mind. Defending LLM integrations against prompt injection covers the practical steps.
Useful metadata fields:
source_id and chunk_index (for traceability and citations)
title and section_path
doc_type, language, product or region as relevant
effective_date, version and status
access_groups or an equivalent permission tag
content_hash (to detect changes and skip re-embedding)
How to test chunking, step by step
You can argue about chunk size indefinitely. Testing ends the argument.
Step 1: build a question set. Collect 30 to 100 real questions your users ask or would ask, and for each note which document and passage contains the answer. Real questions matter: they are phrased differently from the documents, which is exactly the challenge. Build the set carefully, following the approach in building a golden test set.
Step 2: measure retrieval alone. For each question, run retrieval and check whether the correct passage is among the top k results. Common measures:
Hit rate at k: the share of questions where the right chunk appears in the top k.
Mean reciprocal rank (MRR): the average of 1 divided by the rank of the first correct result, which rewards putting the right chunk near the top.
Precision at k: how many of the top k are relevant, which shows how much noise you pass to the model.
This isolates the effect of chunking from the effect of the language model.
Step 3: vary one thing at a time. Try recursive versus structure-aware splitting. Try 300, 500 and 800 tokens. Try 0, 10 and 20 percent overlap. Try with and without header prefixes. Keep everything else fixed and record the metrics for each.
Step 4: inspect failures. For each miss, look at what was retrieved instead and at how the correct passage was chunked. Typical findings: the answer was split across two chunks; the relevant chunk was dominated by an unrelated topic; the chunk lacked the words the question used; a heading was separated from its content.
Step 5: measure end to end. Once retrieval looks reasonable, evaluate the final answers: correctness, groundedness in the retrieved text, and whether the system properly says it does not know when the answer is absent. If retrieval is good but answers are poor, the problem is elsewhere, in the prompt, the model or the context assembly.
Step 6: track cost and latency. Record the number of chunks, tokens per query and time. A slightly better hit rate that doubles the prompt size may not be worth it.
Step 7: keep the test. Re-run it whenever you change the corpus, the chunking or the embedding model. Changing the embedding model means re-embedding everything, and the two must match: vectors from different models are not comparable.
A small script can run this loop. Here is the shape of one in Python:
def hit_rate_at_k(questions, retrieve, k=5):
hits, reciprocal_ranks = 0, []
for q in questions:
results = retrieve(q["question"], k=k) # list of chunk dicts
ranks = [i for i, r in enumerate(results, 1)
if r["source_id"] == q["answer_source"]
and q["answer_text"] in r["text"]]
if ranks:
hits += 1
reciprocal_ranks.append(1 / ranks[0])
else:
reciprocal_ranks.append(0)
return hits / len(questions), sum(reciprocal_ranks) / len(questions)
Run it once per chunking configuration and compare. The answer_text in r["text"] check is crude but effective: it asks whether the chunk actually contains the answer passage, not merely whether it came from the right document.
Vector store considerations that interact with chunking
Chunking decisions interact with how you store and search.
Index type. If you use Postgres with pgvector, its README describes two approximate index types, HNSW and IVFFlat, and states that approximate indexes trade some recall for speed. If your retrieval tests show misses, check whether the index, not the chunking, is dropping relevant results, by comparing against exact search on a sample. For small collections, exact search is often fast enough and removes that variable.
Distance metric. Use the metric appropriate to your embedding model, and be consistent between indexing and querying. pgvector supports several, including cosine distance, inner product and L2 distance.
Filtering. Combine vector similarity with metadata filters (region, product, permission group). Test how filters interact with approximate indexes, since restrictive filters can leave fewer results than you asked for.
Scale. More chunks means more storage and slower searches; keep an eye on it as the corpus grows. Vector databases for RAG compared helps choose a store, and embeddings API integration explains how to generate the vectors.
When similarity search alone is not enough
Embeddings are good at meaning and weaker at exact strings. A query for a specific error code, part number, person's name or clause reference may not retrieve the right chunk by semantic similarity alone, because the embedding captures the general topic and not the precise token. If your users search for identifiers, consider hybrid retrieval: combine keyword search (for exact terms) with vector search (for meaning) and merge the results. Many databases, including Postgres with its full-text search alongside pgvector, can support both in one place.
Chunking affects this too. Keyword search works best when the chunk actually contains the identifier and its surrounding explanation; if identifiers live only in table cells or file names that you dropped during extraction, no retrieval method will find them. When you inspect retrieval failures, ask whether the query was a "meaning" query or an "exact term" query, because the fix differs.
Another common addition is a re-ranking step: retrieve a generous set of candidates (say twenty), then use a more precise model to reorder them and pass only the best few to the language model. This can compensate for chunking imperfections, at the price of extra latency and cost. As with everything else here, add it because a measured gap justifies it, not because it appears in a diagram.
Chunks and citations
If you want your system to show sources, chunking decides what a citation can point to. A chunk that spans three unrelated topics produces a vague citation. A chunk that maps cleanly to a section, with a stored page number or heading, produces a citation a user can check in seconds.
Store the location metadata at indexing time: document identifier, page or section, and character offsets if you want to highlight the exact passage. Design chunks so that each corresponds to something a person could navigate to. This is what makes answers verifiable, a theme covered in making RAG answers verifiable.
Keeping the index current
Documents change. A chunking pipeline needs an update strategy or the index will quietly go stale.
Detect changes with content hashes, so you re-embed only chunks that actually changed.
Re-chunk whole documents when they change, rather than patching individual chunks, since edits move boundaries.
Delete removed content. Orphaned chunks from deleted documents will keep being retrieved.
Version your chunking configuration. If you change the splitter, record which configuration produced which chunks, and plan a full re-index rather than mixing styles.
Schedule and monitor ingestion. A failed nightly job can leave the index weeks behind without anyone noticing.
A worked example: an internal policy handbook
Imagine a 120-page company handbook, exported to PDF, that employees want to query.
First attempt. Fixed 1,000-character chunks with 200-character overlap. Testing on 40 real questions shows the correct chunk in the top five for only some of them. The misses cluster around policies that have exceptions: the rule is in one chunk and the exception, separated by a page break in the PDF, in the next. Headers and footers ("Confidential – Page 47") pollute many chunks.
Fix 1: clean extraction. Strip repeating headers and footers, rejoin hyphenated words, and preserve headings. Hit rate improves noticeably, because the chunks are cleaner and headings are present.
Fix 2: structure-aware splitting. Split on section headings, prefix each chunk with its heading path, and keep sections under a size cap by splitting on paragraphs. Rules and their exceptions now tend to stay together, and heading prefixes make embeddings reflect the topic.
Fix 3: metadata. Tag each chunk with the policy's effective date and the department it applies to, and filter out superseded versions.
Fix 4: small-to-big for the hardest section. For the benefits section, where answers depend on neighbouring paragraphs, match on paragraphs but return the whole subsection.
Measure. After each change, re-run the 40 questions and record hit rate and MRR, and end-to-end answer correctness on the subset with known answers. Track the tokens sent per query. Keep the version that improves correctness without ballooning the prompt.
Ship with feedback. Log queries where the system answered "I could not find this," and queries where users marked the answer unhelpful. Add these to the test set. The set grows from real use.
The lesson is that most of the gain came from cleaning, structure and metadata, and none from finding a magic chunk size.
Common mistakes
Choosing a size by folklore. Test on your own documents and questions.
Ignoring extraction quality. Bad text in, bad chunks out.
Splitting tables and code as prose. Handle them as their own content types.
Dropping the heading context. Chunks without scope are ambiguous.
Indexing everything. Boilerplate, duplicates and outdated versions degrade results. Curate the corpus.
Forgetting permissions. Filter by access before the model sees anything.
Never re-testing. Any change to documents, chunking or embedding model can shift results.
Assuming retrieval failed when the model failed, or the reverse. Evaluate them separately.
Over-engineering early. Start with recursive or structure-aware splitting and a test set. Add sophistication only where measurement shows a gap.
A short checklist
Extracted text has been inspected for a sample of documents.
Boilerplate and duplicates are removed.
Splitting respects structure where structure exists.
Chunks carry title and heading path.
Metadata includes source, version, date and permissions.
Tables, code and transcripts have specific handling.
Chunk size and overlap were chosen by measurement, not assumption.
Retrieval is tested separately from generation.
The index has an update and deletion process.
The test set is re-run after every change.
If you would like help evaluating and improving retrieval quality on your own documents, get in touch.
Written by Abhishek Patoliya, Senior n8n Developer & Automation Architect, Surat, India. Verified n8n Creator.