What does an LLM evaluation cost? A budget model for SaaS teams

Short answer: an LLM evaluation costs three things: model tokens for the system under test and any LLM judge, human hours for labeling and calibration, and the engineering time to keep the eval set current. For most SaaS teams the token bill is the smallest line. Human review time is usually the largest, and it is the one teams forget to budget until a release is blocked waiting on reviewers.

This guide gives you a cost model you can fill in with your own numbers, a worked example, and the levers that cut the bill without making the eval less trustworthy.

The four cost lines in an eval run

Every eval run, whether it is a nightly regression check or a one-off model comparison, has the same four components. Write them down separately, because they scale differently.

1. Generation tokens

You have to produce the outputs you are grading. That means running every item in the eval set through the system under test: the prompt, retrieved context, tool calls and the response. If you are comparing two models or two prompts, you pay for both sides. If you sample each item more than once to measure variance, multiply again.

2. Judge tokens

If an LLM grades the outputs, each judgment is another model call. A judge prompt is usually larger than people expect: the rubric, the original input, the reference answer if you have one, and the response (or both responses for a pairwise comparison). A judge call of 2,000 to 4,000 input tokens is common once the rubric is written properly.

3. Human review hours

Humans show up in three places: labeling items the judge cannot handle, a calibration sample that checks the judge agrees with people, and an overlap sample that checks reviewers agree with each other. Price this at the reviewer's loaded hourly cost, not their base salary, and use real seconds-per-item measured on a pilot batch rather than a guess.

4. Upkeep

Eval sets go stale. New failure modes appear in production, features change, and the rubric drifts away from what the product team actually cares about. Someone has to add new cases, retire obsolete ones and re-check the judge after any rubric change. This is engineering time, and it recurs every month the feature is live.

A worked example you can copy

The numbers below are illustrative. Model prices change often, so plug in the current rates from your provider's pricing page and the hourly cost of your own reviewers. The structure is what matters.

Assume a support-answer feature with a 500-item eval set, graded by an LLM judge against a written rubric, and checked by humans on a sample.

• Judge call size: 2,500 input tokens and 300 output tokens.
• Example judge price: $3 per million input tokens and $15 per million output tokens.
• Cost per judgment: 2,500 x $3 / 1,000,000 = $0.0075 input, plus 300 x $15 / 1,000,000 = $0.0045 output, so about $0.012.
• Repeats: 3 judge samples per item to smooth out judge variance, so 1,500 calls, about $18 per run.
• Batch processing: eval runs rarely need answers in seconds. Anthropic's Message Batches API charges 50% of standard prices, and OpenAI's Batch API offers a similar discount. That brings the judge bill to about $9 per run.

Now the human side, using a measured 90 seconds per item and an example loaded cost of $60 per hour:

• Calibration sample: 150 items labeled by a person so you can measure judge-human agreement. 150 x 90 s = 3.75 hours, about $225.
• Reviewer overlap: a second person re-labels 75 of those items to check that humans agree with each other. That is about 1.9 hours, about $112.
• Disagreement review: a lead resolves the items where the judge and humans split. Budget an hour, $60.

Total for the first full run: roughly $9 of judge tokens, plus generation tokens, plus about $400 of human time. The human line is more than forty times the judge line. That ratio is typical, and it is why the cheapest savings rarely come from switching to a smaller judge model.

What changes on the second run

Calibration is mostly a one-time cost per rubric version. Once you have shown the judge agrees with people on your rubric, later regression runs only need a small spot-check sample, perhaps 30 to 50 items. The steady-state cost drops to the token bill plus an hour or so of review, until the rubric or the judge model changes. Then you recalibrate.

Where teams overspend

The same mistakes show up repeatedly when we look at eval budgets.

Grading everything with the most expensive judge

Many checks do not need a model at all. Valid JSON, required fields, length limits, banned phrases, and whether a cited document ID exists can all be checked in code for free. Run deterministic checks first and only send the items that pass them to the LLM judge. We cover this split in more detail in our post on using an LLM judge as a release gate.

Paying for human review without measuring agreement

If two reviewers only agree on 70% of items, adding more human labels does not make the eval more reliable. It makes it more expensive and equally noisy. Measure agreement before scaling review; our guide to inter-rater agreement with kappa and alpha shows how to do it on a small sample.

Re-running the full set on every commit

A 500-item set with three judge samples on every pull request adds up quickly. Split the set: a small, fast smoke set on every change and the full set nightly or before a release. Tie the full run to the events that actually change behavior, like a prompt edit, a retrieval change, or a model upgrade, as described in running LLM evals in CI when models update.

Ignoring the cached prefix

Judge prompts repeat the same rubric and instructions on every call. If your provider supports prompt caching, put the rubric in a stable prefix and the item-specific content after it. Our post on prompt caching cost savings explains the mechanics and the cache-hit conditions to watch for.

An eval set full of easy items

If 90% of items pass on every model you try, you are paying to grade items that tell you nothing. Build the set from real failures instead, using the process in building an LLM eval set from production failures, and retire items that no candidate ever fails.

How to set an eval budget

Start from the decision the eval supports, not from a number of items.

1. Name the decision. Ship or hold a release, pick between two models, or detect a regression. Each needs a different level of confidence.
2. Size the set to the smallest difference you care about. Detecting a 2-point change in pass rate needs far more items than detecting a 10-point change. If you only act on large regressions, a smaller set is enough.
3. Pilot 20 items by hand. Time it. Your seconds-per-item number drives the human budget more than anything else.
4. Calibrate the judge once per rubric version. Then spot-check on each run.
5. Put the token line on the cheapest path that meets your latency. Batch for nightly and pre-release runs; standard calls only for the fast smoke set.
6. Review the budget monthly. Retire stale items, add new failures and check whether the human sample can shrink.

When to bring in outside reviewers

Domain-heavy features, like code review, legal or financial answers, or medical triage, need reviewers who actually know the domain. A generalist labeler will agree with the judge on the easy items and miss the subtle ones, which is the opposite of what calibration is for. If you do not have that expertise in-house, an expert review network is usually cheaper than hiring for it. That is the kind of work we do through Boundev's expert AI evaluation services.

FAQ

Is an LLM judge always cheaper than human review?

Per item, almost always, often by two orders of magnitude. But a judge you have not calibrated against people is not a cheaper version of human review. It is an unverified one. Budget the calibration sample as part of the judge's cost.

How many human-labeled items do I need to calibrate a judge?

There is no universal number. A practical starting point is 100 to 200 items covering your main failure types, enough to see agreement per category and not only overall. If agreement is low in one category, add items there rather than across the board.

Should I use the same model as the judge and the system under test?

It is cheaper to set up, but models can favor their own outputs. For model-comparison evals in particular, use a judge from a different model family or validate the judge against human labels on both candidates.

Does batch processing slow down my release?

Anthropic says most batches finish in under an hour, with a 24-hour maximum. That is fine for nightly and pre-release runs. Keep a small smoke set on standard calls for fast feedback during development.