Zovda
Identifier stability is the least glamorous property of a dataset and the one that decides whether longitudinal analysis is possible at all.
If an item's id changes when it is edited, every join you built yesterday returns nothing today, and the drift looks like churn.
Three kinds of change
- The item is edited, and the platform issues a new id
- The item is deleted and restored, and comes back as a new object
- The item is migrated between systems, and the id is rewritten in bulk
Each produces the same symptom: a row that should have matched does not.
What to store
- The native id, as returned
- A content fingerprint, computed on normalized text and author
- The first time the item was seen and the last
- A cluster id, assigned when two records are judged to be the same item
The cluster id is the thing you join on. It survives the renames because it is yours, not theirs.
Say how the match was made
An exact fingerprint match and a fuzzy one are different levels of confidence. Store which rule fired, so a later reviewer can decide whether to trust the cluster.
Merging two real posts is worse than missing a merge, so keep the threshold conservative.
Audit the clusters
Sample a hundred clusters a quarter and read them. Duplicates that should have merged, and merges that should not have happened, both show up immediately to a human and never in a metric.
Zovda assembles these collections through Datuva, where the native id and the cluster id both survive export. The social media data collection shows the fields.
Tell us what you need to track over time. We reply within one business day.