Zovda
Every team has a data mixture. Few can tell you why the numbers are what they are. The ratio was set once, in a spreadsheet, carried over from a paper, and has been inherited by every run since.
A mixture is not a configuration detail. It is a claim about how much each kind of text is worth per token, and most teams have never tested that claim.
A ratio is a set of bets
"Coding at 30%, web at 50%, math at 20%" is three bets stacked on top of each other, each one asserting that the marginal token from that bucket helps more than the next-best alternative.
Made explicit, those bets can be tested. Left implicit, they get inherited, and nobody remembers what evidence ever supported them.
Deduplicate before you tune the ratio
Tuning a mixture on duplicated data is tuning a number that is already wrong. If one bucket is 40% near-duplicates, its effective share is 40% lower than the label says, and every experiment after that is measuring noise.
Deduplicate first, then measure how much each bucket actually lost. The corrected numbers are usually surprising enough to justify the work on their own.
Synthetic text changes the math
Generated rows are cheap, which makes them easy to over-weight. They also concentrate in whatever the generator does well, so a bucket that looks diverse at the label level can be narrow at the token level.
Track synthetic share separately from the rest, and cap it until you have an eval that can catch the narrowing.
Tune on the eval, not on the loss
Loss goes down when the mixture matches the style of the training text. That is not the question you are asking.
Build the eval first: a held-out set per capability, with the same weighting you actually care about. Then change one bucket at a time and read the eval, accepting that the loss will be a worse guide than usual.
The checklist we run
- Can someone explain, from evidence, why each ratio is what it is?
- Is deduplication done before the ratio is set?
- Is synthetic share tracked and capped separately?
- Is there an eval per capability, not just a single aggregate?
- Does every bucket carry a processing history that survives re-export?
If a supplier cannot answer these, the mixture will still train. It will just be an inherited opinion rather than a measured decision.
Where to start
Start from the capabilities you are willing to trade against each other, and build the eval before touching the ratio. The corpus catalogue shows how the same material is read across training stages, which is useful when a bucket needs to be split by stage rather than by topic.
Zovda assembles these mixed collections through Datuva, so the composition above can be rebuilt from the manifest rather than from a slide.
Tell us the capabilities you are balancing and the volume you need. We reply within one business day, and the field documentation is shared once the scope is confirmed.