Short answer: a coding agent's pass rate tells you how many tasks ended with the provided tests green. It does not tell you how many fixes were right. Published audits of SWE-bench show the gap is real: some passing patches behave differently from the developer's fix, and some are plainly wrong. Before a pass rate drives a model or vendor decision, sample the passes, have an expert check each one against the task, and report an adjusted rate with an interval.
What a pass actually certifies
Most coding benchmarks score a task as solved when the agent's patch makes a set of target tests pass without breaking the others. That is a reasonable, cheap signal. It is also a narrow one. A pass certifies that the patch satisfies the behaviors those tests encode. It says nothing about behavior the tests never exercise, and it assumes the task description was clear enough that only one kind of fix makes sense.
Benchmark maintainers know this. SWE-bench Verified, released by the SWE-bench authors with OpenAI in August 2024, is a 500-task subset that human annotators screened so that problem descriptions are clear, test patches are correct and tasks are solvable with the information given, according to the SWE-bench site. Screening improved the tasks. Later research shows it did not make every pass trustworthy.
What published audits found
Two peer-reviewed studies looked directly at passing patches.
- Wang, Pradel and Liu, "Are 'Solved Issues' in SWE-bench Really Solved Correctly?" (arXiv March 2025, revised September 2025; ICSE 2026 research track), compared plausible patches from three issue-solving tools with the developers' own fixes. They report that 7.8% of patches counted as correct while failing the full developer test suite, that 29.6% of plausible patches behaved differently from the ground truth, and that manual inspection found 28.6% of those divergent patches certainly incorrect. They estimate these weaknesses inflate reported resolution rates by 6.2 percentage points.
- Yu, Zhu, He and Kang, "UTBoost" (ACL 2025), generated extra tests and found 36 task instances with insufficient tests and 345 erroneous patches previously labeled as passing. Correcting them changed 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries.
Two cautions about reading these numbers. They describe specific tools and benchmark versions at a point in time, so they are not a correction factor you can apply to any agent. And "behaves differently" is not the same as "wrong": the Wang study found many divergent patches were similar but different implementations. That is exactly why the final call needs a person who understands the code.
A pass-rate audit in five steps
You do not need to re-review every task. You need a defensible estimate of how much of the reported number holds up.
- Draw a random sample of passing tasks. Fix the sample before anyone opens a patch, so reviewers cannot drift toward interesting cases.
- Re-read the task and its tests. Ask whether the task was specified well enough to have one intended behavior, and whether the tests would fail an obviously wrong fix.
- Compare behavior with the reference fix where one exists. Try inputs near the edges of the issue and look for behavior the patch changed outside it.
- Record one verdict per pass, with the evidence that decided it: correct, acceptable variant, incorrect, or task flawed. A task-flawed item is a benchmark problem, not an agent failure, so fix or drop it rather than counting it against the agent.
- Report the adjusted rate with an interval, and state what the audit did not check.
Our guide to what passing tests leave unanswered covers steps 2 and 3 for a single patch in more detail, and choosing evaluation samples covers step 1.
A worked example
This example is illustrative; the numbers are invented to show the arithmetic, not taken from any team or benchmark.
An internal suite has 200 bug-fix tasks. The candidate agent passes 120, a reported pass rate of 60%. The team samples 40 of the 120 passes. Review finds 31 correct, 5 acceptable variants, 3 incorrect and 1 task flawed.
| Verdict | Count | Counts as a real pass? |
|---|---|---|
| Correct | 31 | Yes |
| Acceptable variant | 5 | Yes |
| Incorrect | 3 | No |
| Task flawed | 1 | No; repair the task |
36 of 40 sampled passes hold up, 90%. A 95% Wilson interval for 36 out of 40 runs from about 77% to 96%. Multiplying through, the adjusted pass rate is about 54%, with a plausible range of roughly 46% to 58%. The headline 60% was not a lie, but it was the top of the range, not the middle.
The useful output is not just the smaller number. It is the three incorrect patches and the flawed task, each with a written reason. Those tell you what kind of mistake the agent makes and which tests need strengthening before the next comparison.
When this audit matters most
- Two agents or models are within a few points of each other. A gap smaller than the audit interval is not a decision.
- The pass rate will gate a release, a contract or a migration to a new model.
- Your tasks come from your own repositories, where nobody has screened the tests the way SWE-bench Verified was screened.
- The agent is new to the codebase, and you have no history of how its passes fail.
If you only need a rough trend line across nightly runs, a full audit each time is overkill. Audit when the number is about to change a decision.
Limits of a sampled audit
A sample of 40 tells you about the population of passes it was drawn from, and only within its interval. It does not tell you about the failures, which need their own review if you suspect the tests reject correct fixes. Reviewer judgment can also disagree; for verdicts that matter, use two reviewers on at least part of the sample and resolve disagreements explicitly, as described in our guide to reviewer disagreement. Finally, an audit checks the patches you supplied. It is not a security assessment and does not prove the agent will behave the same on a different codebase.
Next step
Start with the evaluation guide to frame the decision your pass rate is meant to inform, and read how reviews are structured in our methodology. If you want experienced engineers to review a sample of your agent's passing patches, tell us what decision you are making.