FOR AI TEAMS

Human evaluation
for AI-generated code.

Bring software experts into code review, model-response comparisons and rubric development. Get documented judgments and visible disagreement to inform your engineering decisions.

Discuss your evaluation
Software engineers discussing model output quality

From evaluation brief to useful evidence.

01

Share your goals

Share what you want to evaluate. A short outline is enough to start the conversation.

02

Align with our team

We review your request, discuss project fit, and agree on the review approach before approving access.

03

Start your program

Agree on the scope, review criteria, and deliverables with our team. Project access follows approval.

PURPOSE-BUILT EVALUATION

Go beyond
a single score.

See the reasoning behind model performance. Connect each judgment to its artifact, reviewer, and rubric.

01

Code and patch evaluation

Review correctness, maintainability, instruction following, and security.

02

Blinded pairwise comparison

Compare two model responses with independent expert judgments.

03

Structured evaluation data

Bring accepted records into your analysis with CSV and JSONL exports.

Illustration of an annotated code review, a magnifying glass, and a blue pencil on a dark desk
EXAMPLE REVIEW4.5/ 5

Across four criteria

EXPERT JUDGMENT

The detail behind
the decision.

Every score has context: the artifact, the rubric, and the reviewer’s reasoning.

See the score breakdown
Correctness4 / 5
Instruction following5 / 5
Code quality4 / 5
Security5 / 5

4 + 5 + 4 + 5 = 18 out of 20, or an average of 4.5 out of 5.

Illustrative example · Not a published evaluation result.

QUALITY THROUGHOUT THE WORKFLOW

Independent review.
Visible disagreement.
Decisions with a record.

Explore our evaluation methodology

What your AI evaluation should establish

Choose the review around a decision: accept a generated patch, compare model versions, or improve the criteria used to judge an answer.

Your questionUseful inputsEvidence to agree in the scope
Does this code meet the requirement?Code or diff, task requirements, relevant context and supplied test evidence.Criterion-level findings, code references, severity and unresolved questions.
Which response better solves the task?The same prompt and context, candidate responses, and acceptance criteria.Blinded preferences, reasons, ties and recorded disagreement.
Are reviewers using the rubric consistently?A draft rubric and representative examples, including ambiguous cases.Calibration notes, unclear criteria and proposed score anchors.

Review the evidence behind a score

The agreed output can include item-level reasons, artifact references and CSV or JSONL fields for your analysis. A summary should state the sample’s limits and unresolved disagreements. See the worked example report before deciding what your team needs; it is an AI-assisted illustration, not a completed independent review.

Where expert review fits

Human review can examine requirement interpretation and trade-offs that an automated score leaves unclear. It complements your tests and operational measurements. The current service reviews supplied artifacts and evidence; hosted execution of client code is outside scope. An evaluation does not certify software as secure or defect-free.

Prepare a useful first enquiry

Share the decision, programming languages, approximate sample size, deadline and access restrictions. Keep confidential code and data out of the initial form. We agree artifact handling, reviewer requirements, delivery scope and fees before starting.

Discuss your code evaluation

See the workflow for yourself.

Discuss your evaluation

START WITH A DEFINED QUESTION

Shape your first evaluation pilot.

See the inputs, deliverables, and scope we’ll agree together.

Explore the pilot ↗