Zovda
Chinese text has no spaces, so someone has to decide where words end. That decision is usually made once, buried in a preprocessing script, and inherited by every downstream step.
It is a classification decision disguised as formatting, and it changes deduplication, retrieval, and what a model counts as the same sentence.
Ambiguity is normal, not an error
A string can segment two ways and both can be right depending on context. A corpus that stores one segmentation as fact teaches the model that one reading is the only reading.
Keep the raw text as the primary field and the segmentation as a derived, replaceable layer with its own version.
Characters, words and both
Character-level models handle unknown words better. Word-level models handle meaning better. Most production work wants both views, and the corpus should make both available without re-processing.
Store character offsets, not only token strings, so a change of dictionary does not invalidate every annotation.
Normalisation needs a record
Full-width versus half-width, traditional versus simplified, and punctuation variants all appear in real data. Each choice is legitimate; leaving the choice undocumented is not.
We keep a normalisation profile per collection and never mix profiles inside one training run.
The checklist we run
- Is the raw text preserved alongside any segmentation?
- Do annotations use character offsets rather than token ids?
- Is the normalisation profile recorded per collection?
- Can traditional and simplified material be separated?
- Does deduplication run on normalised text with a documented profile?
If a supplier cannot answer these, the corpus is a token stream that cannot be re-segmented. Zovda keeps the raw layer first and builds Datuva-ready derivatives on top.
Where to start
Start from the task: retrieval, classification, or generation each prefers a different view of the same text. The corpus catalogue shows where Chinese families sit.
Tell us the task and the volume. We reply within one business day.