Zovda
Two annotators read the same pair of answers and disagree. Most teams treat that as noise to be reconciled, or worse, to be dropped. It is usually the most informative row in the file.
Preference data is not a survey. It is a record of where judgement is genuinely contested, and the contested cases are what shape a model's behavior under pressure.
Ties are not missing data
A tie means the two answers are close enough that the rubric could not separate them. That is a fact about the task, not a failure of the labeler. Dropping ties teaches the model that every comparison has a clear winner, which is the opposite of what deployment looks like.
Keep ties as a third label, and track how often they appear. A rising tie rate is usually the first sign that a rubric has stopped discriminating.
A rubric beats a vibe
"Which answer is better" produces a label. "Which answer follows the instruction, stays on topic, and does not introduce unsupported claims" produces a decision procedure that another annotator can reproduce.
Write the rubric before the first row. Then test it: hand the same pair to two annotators independently and measure the agreement. If agreement is near chance, the rubric is the problem, not the annotators.
Length bias is the default, not the exception
Given two answers of different lengths, people pick the longer one far more often than they should. The model learns this within one epoch.
Counter it in the collection: balance the preferred side by length, and include pairs where the shorter answer is clearly better. Then measure whether the trained model still prefers length after tuning.
Disagreement is a calibration signal
Where annotators split, the model should be less confident. If your pipeline throws those rows away, you lose the only examples that teach hesitation.
The practical version: keep both labels, keep the rationale, and let downstream training decide whether to use the row for preference learning or for a confidence objective.
The checklist we run
- Are ties kept as a label rather than resolved by majority vote?
- Is there a written rubric that a new annotator can apply without asking?
- Do length and preference correlate in the training set?
- Is inter-annotator agreement measured, not assumed?
- Are rationales stored alongside the choice?
If a supplier cannot answer these, the pairs will still train a model. It will just learn the annotator's habits instead of the intended behavior.
Where to start
Start from the decisions your product actually makes, and build pairs around the ones where two reasonable people could differ. The corpus catalogue shows how preference rows sit alongside supervised rows at each training stage.
Zovda runs these annotation programs through Datuva, where every disagreement above stays as a row instead of being averaged away on a dashboard.
Tell us the decision you are trying to shape and the volume you need. We reply within one business day, and the field documentation is shared once the scope is confirmed.