Zovda
A rubric is not a description of quality. It is a machine for producing the same decision twice, from two different people, without a conversation.
When raters have to meet to agree, the meeting is doing the work the rubric was supposed to do, and it does not scale past the pilot.
Criteria, not adjectives
"Clear and well organized" is an adjective. "The answer states the conclusion in the first two sentences" is a criterion. Only one of them can be checked by a person who did not write it.
Rewrite every line until it names something observable in the output rather than a quality you would like it to have.
Order the criteria
When two criteria conflict, the rubric has to say which one wins. Length against completeness is the usual conflict: a short answer that misses a case, a long one that covers it.
State the tie-break explicitly, or raters will resolve it by taste, consistently enough to look like agreement and differently enough to matter.
Anchor each level with an example
For every level of every criterion, keep one real output that sits at that level. Raters calibrate against examples far faster than against prose.
Rotate the anchors. An anchor that has been in the file for two years is measuring two years ago.
Test on the boring rows
Everyone tests a rubric on the interesting cases. The disagreements usually come from the boring ones, where raters assume the answer is obvious and therefore do not apply the rules.
Zovda runs these rubrics through Datuva, where each rating keeps its criterion and its anchor, so a disagreement can be traced to a line rather than a mood. The question bank collection shows graded answers under this kind of scheme.
Tell us what you are grading. We reply within one business day.