Zovda
Generating evaluation traffic with a model is tempting: unlimited questions, no privacy work, no waiting for users. It is also a way to test a model against its own expectations.
A synthetic user asks the question the assistant is ready for.
The blind spot is structural
A model playing a user has read the same training material as the model being evaluated. It produces questions that are well formed, on topic, and courteous, because that is what the material looks like.
Real users are short, impatient, and often ask for something the product does not do.
Where synthetic still helps
- Volume for regression checks on whether the format is followed
- Adversarial variants of an existing real question
- Coverage of languages and registers that are hard to collect
- Stress cases that a human would find uncomfortable to write
Those are worthwhile, and they are not a substitute for traffic.
Keep the origin field
Every generated question should say which model wrote it and from what seed. A benchmark whose rows cannot be traced will drift silently as the generator changes.
Mix, then measure the mix
A set that is mostly synthetic measures the generator's taste. Report the share, and keep the real rows in their own slice so the two numbers stay separate.
Zovda builds evaluation sets through Datuva, where each row records whether it came from traffic or from a generator. The corpus catalogue shows where those rows sit.
Tell us what you cannot collect yourself. We reply within one business day.