01 / Independent expert review
Start with an independent view.
Reviewers complete tasks independently. They do not see each other’s answers before submission, helping preserve distinct technical judgments.
The supplied artifact, task brief, and rubric give each reviewer a common basis for their assessment.
A shared starting point
Each expert sees the same task material and scoring criteria. Their reasoning is recorded before they see another reviewer’s judgment.
02 / Disagreement rules
Make uncertainty visible.
Numeric scores normally escalate when a reviewer’s score differs from the median by two or more points. Categorical disagreement escalates by default.
Projects can configure these rules to reflect the task and the cost of an incorrect judgment.
An example, not a benchmark result
Scores of 5, 4, and 2 have a median of 4. The score of 2 is two points away, so the task escalates for senior review.
Explore how disagreement is resolved →03 / Hidden gold tasks
Keep quality in view.
Verified control items can be included within the task stream to monitor evaluator accuracy. Reviewers are not told which tasks are controls.
Control performance is one quality signal alongside reviewer agreement and project history.
Use more than one signal
A control item helps assess a known case. Review quality also depends on the explanation, agreement patterns, and the context of the project.
04 / Senior adjudication
Resolve the reasoning.
Escalated tasks receive an independent senior review before a final label is produced. The adjudication considers the artifact, rubric, and submitted explanations.
The decision and its rationale remain part of the evaluation record.
Keep the evidence together
The final judgment stays connected to its artifact, rubric, reviewer explanations, and adjudication rationale.