Zovda
An instruction set can look immaculate — clean prompts, careful answers, consistent tone — and still make the model worse. The problem is rarely the answers. It is that the set quietly teaches a style instead of a behavior.
That is the strange property of instruction data: it is the easiest corpus to collect and the easiest to get subtly wrong.
Style leaks faster than knowledge
A set written by one team tends to sound like one team. The openings repeat. The hedging repeats. The model learns the voice long before it learns the task, because the voice is what every single row has in common.
The fix is not to write more carefully. It is to vary who wrote what, and to check whether a classifier can identify the task family from the first sentence. If it can, the model probably can too.
Duplicates are the expensive kind of easy
Near-duplicates hide in paraphrase. Three prompts that ask the same thing in slightly different words will train three times on one behavior and none on the ones you skipped.
Deduplicate at the intent level, not the string level: normalize the prompt, strip the domain nouns, and cluster what remains. What survives should be a set of distinct behaviors.
Length is a label
If every long prompt has a long answer and every short prompt a short one, the model learns the pairing and not the reasoning. Mix lengths deliberately: short prompt with a long answer is often the row that carries the actual behavior.
Held-out prompts must look like real ones
A held-out set written by the same people who wrote the training set measures style consistency. Pull the held-out prompts from real usage — the messy, half-specified requests that arrive in production — or you will tune to a test that no user will ever take.
The checklist we run
- Can a classifier guess the task family from the style of the answer? If yes, vary the writing.
- Are duplicates removed at the level of intent rather than wording?
- Do prompt and answer lengths vary independently?
- Do held-out prompts come from a different origin than the training rows?
- Does every row carry a task family and a difficulty label?
If a supplier cannot answer these, the set will still produce a model that sounds polished. Polished is not the same as capable.
Where to start
If you are building the supervised fine-tuning set, start from the behavior you want to change and work backwards to the rows that demonstrate it. The corpus catalogue shows how instruction rows are read at each training stage, and the question bank collection is the closest neighbour when your task is reasoning over questions rather than conversation.
Zovda builds these SFT collections through Datuva, where style and capability tags sit on each item rather than on the dataset, so a ratio can change without re-labelling.
Tell us the behavior you are trying to teach and the volume you need. We reply within one business day, and the field documentation is shared once the scope is confirmed.