Zovda
A monthly refresh feels like appending rows. It is a schema change wearing a costume, because the upstream system that produces the data also changed, and it did not ask.
Nothing breaks when a field is renamed and the loader fills the column with nulls. The training run completes. The result is quietly different.
The quiet break
Three things change on a refresh and only one of them is rows:
- New rows, in the same schema, which is the case everyone plans for
- New values in an existing field, which is the case that breaks the mix
- A changed definition, such as a category that now includes something it used to exclude
The third is invisible in row counts and decisive in behavior.
Pin the schema, version the content
Treat the field list as a contract with a version number. When the upstream adds a value, that is a new version, not a Tuesday.
Keep the old version readable for as long as anything downstream might still be running on it. A pipeline that can only read the newest schema cannot reproduce last month's run.
Backfill is a separate job
Filling history with the new definition creates a set that looks consistent and is not, because the old rows were produced under rules that no longer exist.
Either backfill with the original rules, and label it as such, or leave history alone and start the new definition at the date it was introduced. Write the choice down.
What to log on every refresh
Rows added, rows removed, distinct values per category with the delta, and the date the upstream changed. Four lines that turn a mystery into a diff.
Zovda maintains these collections through Datuva, so a refresh keeps its own record and a deleted item upstream does not become a hole downstream. The corpus catalogue lists the families that refresh monthly.
Tell us how often your data has to move. We reply within one business day.