Start with what was actually checked
A passing suite tells you that the checks it ran passed in that environment. It does not tell you that every important requirement was checked. The useful next question is which behaviors and constraints remain unsupported by the supplied evidence.
EvalPlus extends the tests in HumanEval and MBPP to examine correctness more rigorously. It is a concrete example of why the choice of checks matters. It does not establish that any finite suite covers every production concern.
Read the requirement before the patch
Write down the intended behavior, boundary conditions, and prohibited changes. For an expiry check, specify whether equality is valid. For pagination, specify the final page and invalid input behavior. Without these anchors, reviewers can give different scores while applying different unstated rules.
Then link each supplied test result to the requirement it supports. Record the command, environment, artifact revision, and outcome when those are available. If execution was not supplied or performed, say so.
Separate the unanswered questions
A patch can be easy to read and still be wrong. A broad average can hide a critical defect. Agree any blocking conditions before scoring and report those conditions directly.
- Correctness: does the change implement all the stated behaviors?
- Instruction following: did it respect the requested scope and constraints?
- Maintainability: is the implementation understandable in its surrounding code?
- Risk: are there material concerns that need additional evidence or a specialist review?
An example boundary
Consider a session with expiresAt=100 at now=100. If the requirement says it is expired, a greater-than-or-equal comparison is wrong even when the available happy-path checks pass. The relevant finding is the missing boundary behavior, supported by the exact input and requirement.
Use this reasoning to select the next evidence to collect. It might be an additional client-run test, a clarification from the requirement owner, or a focused expert review. Avoid turning an untested concern into a confirmed defect.
A practical review record
- Requirement and artifact revision.
- Checks supplied and what they cover.
- Finding, cited evidence, and severity under the agreed rubric.
- Missing evidence and the next action needed.
- A conclusion limited to the reviewed scope.