Zovda
Retrieval and matching models are usually judged on a score that a random negative makes look excellent. The same model then fails on the near miss that real traffic is made of.
The difference between a useful set and an impressive one is where the negatives come from.
Random negatives flatter the score
If the wrong answer shares no words with the query, the model only has to notice that the topics differ. That is a vocabulary test, not a judgment test.
A negative that shares the topic, the phrasing, and one wrong detail is a different problem, and it is the problem users actually create.
Mine from the model's own errors
The cheapest source of hard negatives is the current model. Take the cases it ranked highly and got wrong, confirm them by hand, and add them.
This also keeps the set current: as the model improves, the negatives it finds get harder, and the set ages in the right direction.
Keep the hard ones honest
A hard negative is still a negative. If the annotator has to argue that it is wrong, it is probably a labeling error waiting to be found.
Require a one-line reason per hard negative. Rows without a reason are how a set quietly fills with items that are actually correct.
Track difficulty, not just count
Report the distribution of negatives by how close they are, not only the total. A set where a tenth of the negatives are hard behaves differently from one where half are.
Zovda builds these collections through Datuva, where each negative keeps the reason it was judged wrong. The image-text collection applies the same idea to pairs.
Tell us what your model has to distinguish. We reply within one business day.