01 / INDEPENDENT JUDGMENTS
Keep each perspective intact.
Reviewers submit an assessment before seeing other reviewers’ answers. Each judgment is tied to the supplied artifact and the same rubric, with an explanation that can be examined later.
This preserves the reasoning that led each expert to their conclusion, including the differences that a final score might hide.
02 / ESCALATION
Define when to look closer.
The default numeric rule flags a score when it differs from the median by two or more points. Categorical disagreement also escalates by default. These thresholds can be configured for the project.
| Reviewer | Score | Distance from median |
|---|---|---|
| Reviewer A | 5 | 1 point |
| Reviewer B | 4 | 0 points |
| Reviewer C | 2 | 2 points · Escalate |
The median is 4. Reviewer C’s score meets the default escalation threshold.
03 / ADJUDICATION
Return to the evidence.
A senior reviewer assesses the task independently and considers the artifact, rubric, and explanations before producing the final label. The decision and rationale remain part of the evaluation record.
A useful final judgment preserves the reason behind the decision.Read the full methodology