Zovda
Every annotation project starts with a document that describes the categories. Most of those documents describe what the categories mean and never describe what to do when a case sits between two of them.
The second part is the whole job.
Write the decision, not the definition
"Mark as promotional if the content promotes a product" is a definition. It does not settle the case where the product is mentioned once in passing, or where the creator is paid but never names the brand.
A guideline is a set of decisions: if this, then that, even when the sentence reads like the other category.
Include the ugly cases on purpose
The first hundred examples in a guideline should be the ones that took an argument to settle. Put the argument in the document, next to the decision, and note which way it went and why.
Annotators do not need reassurance. They need precedent.
Test the guideline on yourself
Before the project starts, label 30 items alone, twice, a day apart. The items you disagree with yourself on are the ambiguous ones. Rewrite those rules before anyone else reads them.
This takes an afternoon and saves a week of rework.
Version it like code
Guidelines change. When they do, the change needs a date, a reason, and a list of the items already labeled under the old rule. Otherwise a disagreement six months later turns out to be two people following two documents.
Zovda runs these projects through Datuva, where every disagreement stays as a row instead of being averaged away on a dashboard. The question bank collection is one place where this discipline shows up in the output.
Tell us the categories you are trying to separate. We reply within one business day.