Zovda
Deduplication is treated as housekeeping. It is closer to editing: removing a row changes how much weight every remaining row carries.
A set described as 40% instruction data is not 40% instruction data if a third of that portion is one template with the nouns swapped.
Exact match is the easy tenth
Hashing whole documents catches the copies that were copied. It does not catch the rewrite, the re-render, the same page with a new header, or the same question asked in four phrasings.
The easy part is also the part everyone reports, which is why "we deduplicated" often means very little.
Three cheap signals
- Shingle overlap on normalized text, to catch the near copy
- Embedding distance within a cluster, to catch the paraphrase
- Structural keys, such as the same title with a different body
Run all three on a sample first. Each one has a false positive rate that grows with short text, and short text is most of what a chat corpus contains.
Dedup changes the mix, so re-measure
If a category loses a fifth of its rows to duplicates, its share of the training set drops without anyone deciding it should. Recompute the composition after the pass, not before.
Keep the duplicate index. When a reviewer asks why a run changed, the answer is usually in that file.
What to keep on purpose
Not every repetition is noise. A canonical answer may deserve to appear more than once; a rare edge case may deserve a deliberate oversample. The difference is intent, and intent has to be written down.
Zovda assembles these collections through Datuva, where the pass and its index stay with the set, so the composition can be rebuilt rather than argued about. The corpus catalogue shows the families this applies to.
Tell us the mix you are targeting. We reply within one business day.