阅读中文版

Nine Languages, Eight of Them Thin

Filtering raw streams into one consistent training set

A dataset marketed as nine languages is often one language with eight appendices. The row count looks healthy because the table sums everything.

The users outside the majority language experience the appendices as the product.

Count speakers, not rows

Five thousand rows in a language with ten million speakers and five thousand in one with eight hundred million are not the same amount of coverage. The second is noise with a language tag.

Report per-language volume next to the number of speakers the language reaches, and the gap becomes visible in one table.

What per-language coverage needs

  • Rows written by a native speaker, marked as such
  • A separate check that the domain matches, not only the language
  • Formatting conventions that differ by locale: dates, names, numbers
  • A per-language rate of rows that failed quality review

The third is the one that breaks quietly. A correct sentence with the wrong date convention is a wrong sample that no language check will catch.

Native and converted rows are different products

Rows written in the language and rows produced from another language behave differently: the second carries the source language's rhythm, and a model trained on them will sound slightly off in a way users notice immediately.

If you use both, label them separately and set a floor for the native share. Mixing them without a field means you cannot answer the question later.

Set a floor per language

Decide the minimum volume below which a language is not worth shipping, and hold it. A language with a hundred rows is worse than no language at all, because it produces confident mistakes.

Zovda builds these collections through Datuva, where language, native status, and quality outcome are separate fields. The audio collection applies the same discipline to speech.

Tell us the languages your users actually write in. We reply within one business day.

Tell us which languages you need

Related insights

Training data you can put into a run

Tell us what you are training and which modalities matter. We reply within 1 business day.

Start a trialSee the corpus catalogue

Request a trial

Tell us what you need. We reply within 1 business day.