Zovda
A benchmark is only a benchmark while you have not trained on it. The moment a test item enters the training set — directly, paraphrased, or as a copy in another language — the score stops measuring capability and starts measuring recall.
The uncomfortable part is that this happens by default. Test sets are published, scraped, and mirrored; training piles are assembled from the same open text. Contamination is not an accident a careful team avoids. It is the baseline you have to actively work against.
Contamination rarely arrives as a whole test file
Nobody pastes a benchmark into a training set on purpose. It arrives as a paragraph quoted in a blog post, as a solved example in a tutorial, as a forum post where someone works through the exact question.
Whole-file checks will miss all of it. What catches it is a per-item overlap check: shingles for exact text, embeddings for paraphrases, and a separate pass for the answer alone.
Build the held-out set like a product
The held-out set deserves the same design attention as the training data. That means a spec: which capability each item tests, how difficult it is, how it is scored, and what a reasonable model should get wrong.
A set assembled from whatever was left over after training tells you the model memorized less of the leftovers.
Check overlap, then check it again after augmentation
Augmentation is where contamination sneaks back in. Render the training set into another language and any benchmark inside it comes along. Paraphrase it, and exact-match checks stop working.
Run overlap against the augmented corpus, not just the original. It is the same check, applied to the version you are actually training on.
Split by time when the task allows it
For anything with a natural chronology — code, news, product data — a time split is the strongest protection available. Train on what existed before a date, evaluate on what appeared after. The model cannot have seen the future.
Where a time split is impossible, hold back entire collections rather than random rows, and document which ones.
The checklist we run
- Is every eval item checked against the training corpus at the paragraph level?
- Are paraphrases and language-converted copies covered by a separate near-duplicate pass?
- Is the answer alone checked, not just the question?
- Is there a private holdout that has never been published anywhere?
- Does the manifest state the split rule, not just the counts?
If a supplier cannot answer these, the number they report will be real. It just will not mean what you think it means.
Where to start
Start by deciding which capabilities you refuse to compromise on, then build the held-out set around those before training begins. The corpus catalogue shows how evaluation material is kept separate from training material across stages, and the question bank collection covers the case where leakage hides inside the question itself.
Zovda builds held-out sets through Datuva, where each evaluation row carries the date it was drawn and the collection it came from.
Tell us what you need to measure and the volume you need. We reply within one business day, and the field documentation is shared once the scope is confirmed.