Zovda
The first join between two datasets takes an afternoon. The second one, three months later, takes a week, because by then both sides have changed and nobody wrote down what the first afternoon assumed.
Field mapping is not a spreadsheet exercise. It is a set of decisions about meaning, and meanings drift.
The join looks trivial on day one
On day one, both sides have a post id, a timestamp, a text field, and a count. The join works, the numbers look plausible, and the project moves on.
What did not get decided: whether the timestamp is in the local time of the account or in one fixed zone, whether the count is lifetime or a rolling window, and whether a deleted item is absent or present with a flag.
The fields that decide a join
- The time zone and the format, written down, not inferred from an example
- Whether ids are stable when an item is edited
- Whether counts are cumulative and from when
- Whether an empty field means zero or unknown
- The date the field was introduced
The last one is what makes an old row interpretable. A field that appeared in June has no meaning for a row from March, and filling it with a default invents history.
Where names collide
The same handle across datasets is not the same account. The same display name is not the same person. Brand names collide across languages and verticals, and a naive join quietly merges two audiences into one.
Keep the source id and a confidence field. When the merge is uncertain, store the uncertainty rather than resolving it at load time.
Keep the mapping as a file
Write the mapping as a versioned file, next to the data, with the date and the reason for each change. It is the cheapest documentation you will ever produce and the only thing that makes a rerun possible.
Zovda ships each collection through Datuva with its field mapping attached, so a join made this year can be explained next year. The data research collection shows how these sets are assembled.
Tell us which systems you need to reconcile. We reply within one business day.