Zovda
Retrieval quality is usually blamed on the model and usually caused by the corpus. A dump of documents has duplicates, outdated versions, and passages that were never meant to be read alone.
No ranking model recovers a correct answer from a corpus that contains three contradictory ones.
Three problems a dump creates
Duplicates that split the score between identical passages. Stale versions that outrank current ones because they are longer. Fragments that begin mid-sentence and answer nothing.
Each is fixable at collection time and expensive to fix later.
What a retrieval record holds
- The passage, cut at a semantic boundary and readable alone
- The document id and the version valid at that date
- The section path, so context survives the chunk
- The date the passage became valid and, where known, invalid
- A flag for content that should rank low rather than be removed
That last flag keeps superseded material available for historical questions without letting it win current ones.
Chunks must stand alone
A passage that starts with "as mentioned above" is a fragment. If the cut cannot be moved to a boundary, prepend enough context to make the passage answer a question on its own.
This single rule improves retrieval more than most reranking work.
Evaluate retrieval separately
If the answer is wrong, the first question is whether the right passage was in the top ten at all. Measure that before touching generation.
Zovda builds these collections through Datuva, where each passage keeps its document id and version. The question bank collection shows how answers are tied to the passage that supports them.
Tell us what your system has to retrieve. We reply within one business day.