Zovda
Data plans are written in rows and paid in invoices, and both numbers hide the one that determines quality: how much human attention each hundred rows receives.
A program with the budget for labeling and none for review produces a large set of unchecked labels.
The three lines that matter
- Production: the hours to produce a first pass
- Review: the hours to check a sample and fix what fails
- Rework: the hours to redo what the review rejected
Most plans budget the first and discover the other two.
Review is sampled, not total
Checking every row is rarely affordable and rarely necessary. Checking a random sample at a stated rate, with a stated acceptance threshold, gives you a quality number you can defend.
Write both numbers down before the batch starts.
Rework is a design choice
If the rework rate is high, the guideline is the problem, not the annotators. Budget a guideline revision after the first batch, and expect the rate to drop.
Track cost per accepted row
Divide total cost, including review and rework, by the rows that passed. That number is the only one comparable across batches, vendors, and tasks.
Zovda runs these programs through Datuva, where review results and rework are recorded per batch. The question bank collection shows the output of a well-reviewed program.
Tell us the volume and the quality bar. We reply within one business day.