Zovda
A reward model is a compressed summary of your preferences, and compression keeps whatever is easiest to measure. If length predicts winning in the data, length is what gets kept.
The symptom shows up late, as answers that get longer, more structured, and not better.
The signal you did not intend
Preference data collected without controls usually has three unintended signals: longer wins, more confident wins, and the answer shown in the first position wins more often.
Each one is a shortcut the reward model can learn in a few hundred rows, and each one survives because it correlates with quality often enough to look harmless.
Balance length inside the pair
Do not balance length across the dataset. Balance it inside each pair: the two answers should be close in length, or the length difference should be the deliberate reason one is better.
If a longer answer genuinely wins, write down why. That reason is the feature you want the model to learn, not the word count.
Check with a length-only baseline
Before training, fit a trivial model that sees only length and position. If it predicts the winner well above chance, your set is measuring those two things.
This takes minutes and saves a quarter of tuning a model that is optimizing the wrong target.
Re-audit after every refresh
Preference sets drift toward whatever the current model produces. A set that was balanced last quarter is usually not balanced now, because the new rows came from a model with different habits.
Zovda builds these sets through Datuva, where each pair keeps the reason it was preferred, so the audit has something to check against. The corpus catalogue shows the pairs and stages this applies to.
Tell us what your reward model keeps getting wrong. We reply within one business day.