Zovda
A gold set is the part of the pipeline everyone treats as fixed. It is the part that ages fastest, because the judgment behind each row was made by people who have since learned something.
A stale gold set does not fail loudly. It produces a number that looks stable and means less every quarter.
What a gold set is actually for
It answers one question: did this change make the output better according to the standard we agreed on. If the standard has moved, the answer is about the old standard.
That is fine for a regression check and misleading for a launch decision.
Sample it from the same stream
Gold rows pulled from a curated spreadsheet measure the spreadsheet. Pull them from the same distribution the product sees, with the same filters, so the score reflects the traffic.
Keep the sampling frame in the card. When the frame changes, the number changes for a reason that has nothing to do with the model.
Re-label a slice every quarter
Take a tenth of the gold set, have it labeled again by people who do not see the old labels, and compare. The rows where the two rounds disagree are the ones where the standard moved.
Then do one of two things: fix the row, or fix the guideline. Leaving it alone means the score is quietly averaging two standards.
Report agreement, not only accuracy
A gold set that two experts only agree on 80% of the time puts a ceiling on everything measured against it. Report that number next to the model's, or the reader will assume the ceiling is 100.
Zovda builds held-out sets through Datuva, where each row carries the date it was drawn and the collection it came from. The corpus catalogue shows which families ship with a held-out slice.
Tell us what you are evaluating. We reply within one business day.