Zovda
A dataset is almost always a sample of the common cases, because common cases are what a pipeline collects by default. The product then faces everything.
The gap between the two is where support tickets come from.
The tail is not rare in aggregate
Each individual rare case is rare. Together they are a large share of the questions, which is why a model with good averages can feel unreliable in use.
Count how many requests fall outside the top categories. That number is usually surprising and is almost never in the report.
Deliberately oversample the tail
If the tail is a third of traffic and a percent of your data, the model is calibrated for the middle. Oversample until the tail has enough rows to learn from, and record the oversampling so nobody mistakes the trained distribution for the real one.
Tail cases need their own checks
A tail case often has no single correct answer, only a range of acceptable ones. Build the check accordingly, with a rubric rather than an exact match.
Keep a tail-only evaluation
A slice of nothing but rare cases tells you whether the model improved where it was worst. It will always score lower, and it is the number that matters.
Zovda builds to coverage strata through Datuva, where each batch reports what it covers beyond the common cases. The corpus catalogue shows how coverage is described.
Tell us what your users ask that is unusual. We reply within one business day.