Measuring evaluator reliability
Understanding agreement without treating consensus as the only signal.
REPORTS & NOTES
Notes from
the work.
The methods, questions, and decisions behind expert evaluation. Written to make the reasoning easier to follow.
FEATURED NOTE / EVALUATION DESIGN
What happens
when experts
disagree?
Disagreement can reveal what a single score misses. A closer look at independent judgments, escalation rules, and senior review.
September 2026Read the note
Continue reading
Methods & research directionsExpert evaluation methodology
Independent review, hidden gold tasks, and a traceable final judgment.
Introducing the CodeBench direction
The proposed scope of a benchmark for production-like coding tasks.
Forthcoming
FROM RESEARCH TO PRACTICE
Put expert judgment
to work.
Build an evaluation program around the questions that matter to your team.