Teams come to us with the same complaint. They built a chatbot over their own documents, watched it answer confidently, then caught it inventing a policy that does not exist. The instinct is to blame the model. Almost always, the model is the last thing at fault.

Retrieval-augmented generation is a pipeline, and a pipeline fails at its weakest stage. By the time the language model receives its context, the mistake has usually already been made.

Chunking decides what can be found

Splitting documents every 500 characters is the default in every tutorial, and it is why so many answers arrive half-formed. A table gets severed from its heading. A clause loses the sentence that qualified it. The retriever then returns a fragment that is technically relevant and practically meaningless, and the model fills the gap the only way it can — by guessing.

  • Split on structure — headings, sections, list boundaries — not character counts
  • Keep each chunk's parent heading in its text, so context survives retrieval
  • Overlap chunks slightly, so a sentence spanning a boundary is not lost
  • Store the source and section with every chunk, so citations are possible later

Retrieving more is not retrieving better

Raising the top-k from five to twenty feels like a fix. It is usually a regression. The extra chunks are lower-relevance by definition, and they dilute the context window with near-misses the model must now weigh against the real answer. Precision matters more than recall once you are past a handful of results.

A reranking pass over a small candidate set consistently beats a large unranked one. Retrieve twenty, rerank, pass the best four.

Decide what happens when sources disagree

This is the stage nearly everyone skips. Two documents contradict each other — an old policy and its revision, two regional price lists — and the pipeline silently passes both. The model picks one. It cannot tell you it picked one, because nothing asked it to.

A system that flags a contradiction is more useful than one that resolves it invisibly. Users can handle 'these two sources disagree'. They cannot handle being quietly wrong.

On our own web-scraping RAG build, conflict detection between sources turned out to be the feature people valued most. Not the answers — the honesty about which answers were contested.

Ground the prompt, then verify the output

Instruct the model to answer only from the supplied context and to say plainly when the context does not cover the question. Then check that it obeyed: if the answer contains claims with no supporting chunk, treat that as a failure, not a stylistic quirk.

Answer using ONLY the context below.
If the context does not contain the answer, say so — do not infer.
Cite the source id for every claim you make.

<context>
{{chunks}}
</context>

None of this is exotic. It is ordinary engineering discipline applied to a stage most teams treat as a black box. Do it and hallucination stops being a mystery and becomes a bug with a location.