Zovda
Evaluation sets are built from the same pool as training data, cleaned by the same filters, and reviewed by the same people. Then they are used to claim the model generalizes.
What they measure is performance on the easy half of the distribution, described with more confidence than the sample supports.
Difficulty is a distribution
A set where every question has one clear answer measures recall of patterns. It cannot tell you what happens when the input is incomplete, the question is ambiguous, or two rules conflict.
Add difficulty on purpose: a slice with missing context, a slice with a wrong premise, a slice where the right answer is to ask.
Sample from failures
The best source of hard evaluation rows is production failure. Take the cases users complained about, the ones the model answered confidently and wrongly, and the ones a support person had to fix by hand.
Those rows are already hard, already realistic, and already the ones your stakeholders care about.
Keep a frozen slice
However the set is maintained, keep one slice untouched: same rows, same labels, same order, never refreshed. It will age, and that is the point. It is the only number that can be compared across a year of changes.
Report it next to the refreshed scores, not instead of them.
Report per slice, not per total
A single score hides the trade you just made. Report by difficulty, by source, by language, and by the type of failure the row was designed to catch.
Zovda builds held-out sets through Datuva, where each row carries the date it was drawn and the collection it came from. The question bank collection shows graded items built for this purpose.
Tell us what your evaluation has to catch. We reply within one business day.