Retrieval over versioned corpora

The demo corpus does not exist

Almost every demonstration of retrieval-augmented generation is built on a corpus that behaves impeccably: a fixed set of documents, each internally consistent, none of them contradicting another. Chunk it, embed it, retrieve the nearest few, and the answer falls out.

The corpora people actually have are policies, contracts, standard operating procedures, price lists and technical manuals. They are revised. They are superseded. They are occasionally withdrawn and then quietly reinstated. A document is not a fact; it is a claim with a date attached, and the date is usually the part that matters.

Ask such a system what the notice period is and it will confidently average four versions of the answer, cite the one with the best cosine similarity, and give you something that was true in 2023.

What versioning actually breaks

It is worth being precise about the failure, because it is not one failure but four, and they have different fixes.

• Retrieval returns superseded text that is a perfect semantic match — often a better match than the current version, because revisions tend to add hedging and length.
• The generated answer blends two versions into a third that never existed in any document.
• Citations point at text that is still in the store but is no longer operative, which is worse than no citation: it looks like diligence.
• Nobody can tell afterwards which version the system used, so the answer cannot be audited — and in the settings where this matters most, auditability is the whole point.

Three places to put the version

There are broadly three levels at which version-awareness can live, and they cost very different amounts.

The cheapest is to put it in the chunk text: prepend “Effective 12 March 2024, superseded 4 August 2025” to every chunk and hope the model reads it. This works more often than it deserves to, and fails silently when the retriever returns three chunks with three different headers and the model picks a lane.

The middle option is metadata filtering: store the effective and superseded dates as fields and filter at query time. This is correct whenever the query has a date in it, and most queries do not. “What is the notice period?” carries an implicit “as of today” that nothing in the pipeline is obliged to notice.

The expensive option is to make version a property of the index rather than of the documents in it: one logical record per clause, with a history, so retrieval hits the record and the record decides which text is current. This is closer to how a good CMS models content than how a vector store usually does, and it is the only one of the three that survives a question phrased in the past tense.

A document is not a fact. It is a claim with a date attached.

What we are testing

We are building small evaluation sets out of genuinely versioned material — the kind where the right answer provably changed — and running the three approaches against questions of four shapes: current-state questions, historical questions, “when did this change” questions, and questions whose answer differs depending on a date the user did not supply.

The last category is the interesting one. The correct behaviour there is almost certainly not to answer. It is to notice the ambiguity and ask. Very little in the standard retrieval pipeline is set up to do that, because the pipeline is built to always produce something.