Why Your RAG Chatbot Retrieves the Wrong Chunks (And How to Fix Your Chunking Strategy)
By Abhishek Patoliya · 2026-08-17
A user asks your RAG chatbot a specific question — "What's the cancellation notice period for enterprise plans?" — and it answers with the notice period for a different plan tier entirely, or worse, says it doesn't know despite the answer sitting in your docs. The instinct is to blame the model. Nine times out of ten, the real problem happened earlier: your chunking strategy handed the retriever a piece of text that didn't contain a complete, self-sufficient answer.
How Fixed-Size Chunking Breaks Context
The default most teams reach for is naive fixed-size splitting: cut every document into 500-token blocks, maybe with a little overlap, and embed each one. It's fast to implement and it's also the single most common cause of bad retrieval.
Consider a pricing table. A fixed-size splitter doesn't know a table exists — it counts tokens and cuts. The result is a chunk that contains the row "$249/month, 30-day notice" with no header row above it saying which plan that belongs to. Embedded on its own, that chunk is nearly meaningless: the embedding model has no way to encode "this is about the Enterprise plan" because that information got severed into the previous chunk.
The same failure mode hits FAQ pages (question in one chunk, answer in the next), numbered procedures (step 4 without steps 1–3), and long paragraphs that get sliced mid-sentence right before the clause that actually answers the question.
A Chunking Strategy That Respects Structure
The fix isn't a bigger chunk size — it's chunking that respects the document's actual structure before it ever counts tokens. In practice that means a two-pass approach:
- First, split on structural boundaries: markdown headers, HTML sections, table boundaries, list boundaries. Never let a splitter cut across a table row or a numbered step.
- Second, within each structural section, apply a token-based splitter only if the section itself exceeds your target chunk size (usually 300–500 tokens for short-answer content, up to 800 for narrative documentation).
A rough version of that two-pass logic looks like this:
function chunkDocument(doc, maxTokens = 400, overlapTokens = 50) {
const sections = splitOnHeadersAndTables(doc); // structural pass first
const chunks = [];
for (const section of sections) {
if (tokenCount(section.text) <= maxTokens) {
chunks.push({ text: section.text, heading: section.heading });
continue;
}
// only fall back to token-based splitting inside an oversized section
for (const piece of splitByTokens(section.text, maxTokens, overlapTokens)) {
chunks.push({ text: piece, heading: section.heading });
}
}
return chunks;
}
Notice the heading field carried alongside each chunk — that's not decorative. Prepending the section heading to the chunk text before embedding ("Enterprise plan pricing: $249/month, 30-day notice") gives the embedding model the context it lost when the table got separated from its title, and it dramatically improves recall on questions phrased around that heading's terms.
Overlap and Metadata: What to Attach to Every Chunk
Overlap exists to catch the case where an answer straddles a structural boundary anyway — a 10–15% token overlap between adjacent chunks is enough for most content without bloating your index. But overlap alone won't fix a bad chunking strategy; it just softens the edges of one.
Every chunk should also carry metadata beyond the raw text: source document, section heading, page or anchor, and last-updated date if your content changes. This isn't just for citations in the UI — filtering retrieval by metadata (e.g., "only search the current pricing doc, not the archived 2024 version") often fixes wrong-answer bugs faster than any embedding-model swap.
Measuring Whether Your Chunking Strategy Actually Improved Retrieval
Don't ship a chunking change on vibes. Build a small evaluation set — 30–50 real questions with the chunk ID(s) that should be retrieved for each — and run it before and after any change. Track two numbers:
- Recall@k: for k = 3 or 5, what fraction of questions had the correct chunk somewhere in the top k results?
- Answerability: of the chunks retrieved, how many actually contain a complete answer versus a fragment that needs the missing context from a neighboring chunk?
Teams that skip this step tend to "fix" chunking by shrinking or growing chunk size repeatedly without ever confirming which direction actually helped — because they're only looking at the final generated answer, which the LLM can sometimes paper over with a plausible-sounding guess even when retrieval failed.
When to Reach for Semantic Chunking Instead
For unstructured prose — support tickets, meeting transcripts, long-form articles without headers — structural splitting has nothing to grab onto. That's where semantic chunking earns its cost: embed consecutive sentences, measure the cosine distance between adjacent sentence embeddings, and cut where the distance spikes (a topic change). It's more expensive to compute at ingest time, but for genuinely unstructured content it beats both fixed-size and structural splitting on recall, because it's cutting where the meaning actually changes rather than where an arbitrary token counter hits zero.
Parent-Child Chunking for Documents That Resist Clean Splitting
Some documents resist both structural and semantic splitting — deeply nested technical specs, contracts with cross-referencing clauses, or API references where one endpoint's description spans several paragraphs plus a table plus a code sample. For these, a parent-child chunking strategy tends to outperform either single-pass approach: index small, precise child chunks for retrieval so the vector search matches tightly on what the user actually asked, but when a child chunk is retrieved, expand it to its parent — the full section it belongs to — before handing it to the LLM as context.
This decouples the unit you search on from the unit you generate from. A 100-token child chunk gives you precise embedding matches; the 600-token parent it belongs to gives the LLM enough surrounding context to actually answer completely. The tradeoff is added complexity in your retrieval pipeline — you now need a mapping from child chunk ID to parent section, usually a simple lookup table or a metadata field on each child pointing at its parent's ID — but for documents where neither fixed-size nor purely structural chunking produces clean, self-contained pieces, it's usually worth the extra bookkeeping.
A common variant worth knowing: instead of one parent per child, generate a short LLM summary of the surrounding section and store that summary as the embedded text, while the full section text is what actually gets returned to the generation step. This "summary for search, full text for context" split can outperform plain parent-child expansion on documents where the parent section is long enough that even it doesn't fully capture what makes the child chunk semantically distinct.
A Practical Migration Checklist If You're Already in Production
Changing chunking strategy on a live RAG system isn't a config toggle — it requires re-embedding your entire corpus, which costs money and time proportional to document volume, and it changes what gets retrieved for every existing query pattern your users already rely on. Before starting a migration:
- Build the evaluation set described above first, against your current chunking, so you have a real baseline number instead of a vague sense that "retrieval feels off."
- Re-embed into a separate index or namespace, not in place — this lets you run both old and new chunking side by side against the same evaluation set before cutting over any production traffic.
- Compare recall@k and answerability between old and new before switching, not after — a migration that "feels" like an improvement in a handful of manual spot checks can still regress on query types your evaluation set happens to under-represent.
- Keep the old index available for a rollback window. A chunking regression that only shows up on real user queries, rather than your evaluation set, is far easier to recover from if you can flip back to the previous index immediately instead of re-embedding again under pressure.
Skipping the parallel-index comparison step is the most common mistake teams make here — they re-embed everything, cut over immediately, and only discover the new chunking strategy performs worse on a class of queries their evaluation set didn't cover, by which point the old embeddings have already been deleted to save on storage costs.
Written by Abhishek Patoliya, Senior n8n Developer & Automation Architect, Surat, India. Verified n8n Creator.