Zovda
There are usually three datasets in a project: the training set, the evaluation set, and the traffic. The first two are siblings, and the third is a stranger.
A model that scores well on siblings and behaves oddly on strangers is not a mystery. The distribution was never sampled.
Traffic is a different shape
Requests from users are shorter, misspelled, missing context, and often ask for something adjacent to what the product does. None of that appears in a benchmark built from curated examples.
Sample from traffic, including the ones that were escalated and the ones that were abandoned.
Add the failure modes deliberately
- Incomplete requests
- Requests with a wrong premise
- Requests in a language or register the product does not claim
- Requests that should be refused
Each one is a slice with its own score. An aggregate hides all four.
Keep a production sample frozen
Take a snapshot of a week's traffic, label it, and never change it. It is the closest thing to a fair test of what the product faces.
Refresh it on a schedule by adding a new frozen slice, not by editing the old one.
Report by slice, always
A single number invites the wrong conclusion. Four numbers invite the right question, which is which slice moved.
Zovda builds evaluation sets through Datuva, where each row keeps its origin and its date. The corpus catalogue shows which families ship with an evaluation slice.
Tell us what your traffic actually contains. We reply within one business day.