Load test your AI feature before launch, not after

Most AI features are load tested by their first busy hour in production. That is the wrong place to learn that your provider caps you at a few hundred requests per minute, or that time to first token triples at concurrency your demo never reached. A short, honest load test before launch catches all of it while the stakes are still low.

Load testing an LLM feature is not the same as load testing a normal API. The costs are real per request, the failure mode is a rate-limit wall you do not control, and the metric users feel is not throughput. Here is how to run one that tells you something.

Why LLM load tests are different

A traditional API load test pushes requests per second and watches CPU and error rate. An LLM feature breaks differently. You are calling a provider with its own limits, every call costs money, and a request that returns HTTP 200 can still be too slow to be usable. Three things change the playbook.

First, the wall is external. Providers enforce limits on requests per minute, tokens per minute, and sometimes concurrent connections at once, and those interact in ways you cannot predict without testing. Second, the test itself costs real money, because you are paying per token for every synthetic request. Third, the metric that matters is latency under load, not peak throughput, because a fast-but-wrong-under-load feature still feels broken.

Model real traffic, not "hello"

The most common load-test mistake is sending the same tiny prompt every time. Firing "Hello, how are you?" at your endpoint ten thousand times tells you almost nothing, because real prompts vary widely in length, retrieved context, and output size, and token count drives both latency and cost.

Build a prompt corpus that mirrors production: a spread of short and long inputs, realistic retrieved context if you run RAG, and the output lengths you actually expect. If you already have real traffic from a beta, sample it. If you do not, write 50 to 100 representative prompts by hand. Getting the token distribution right is what makes the latency and cost numbers trustworthy.

Find your rate-limit ceiling before users do

The single most valuable output of a pre-launch load test is your 429 ceiling: the concurrency at which the provider starts refusing requests. Map the exact limits for your API tier, then ramp toward them deliberately.

Start below your known limit and increase load gradually while watching for 429 responses. The point where they begin is your real capacity, not the number on the pricing page. Once you know it, you can decide whether to request a higher tier, add a second provider, or queue and shed load. Whatever you choose, the retry and backoff behavior has to be deliberate rather than a tight loop that makes the wall worse; we covered that in handling LLM rate limits and 429s in production. A feature that hits the ceiling should degrade on purpose, which is the whole point of a designed fallback for when the model fails.

Watch the metrics that predict user pain

Throughput is a vanity number if half those requests miss your latency target. Track the metrics that map to what a user actually feels.

• Time to first token, the delay before the first token arrives. If it crosses about 1 second under load, the feature feels slow or stuck. This is the metric to defend, and our note on time to first token explains why.
• Inter-token latency, the gap between streamed tokens. A common target is under 50ms so the text reads smoothly.
• End-to-end latency, the time to a complete answer, which matters for any non-streaming consumer like an internal API.
• Goodput, the share of requests that meet all your SLOs at once. A useful definition is time to first token under 500ms, inter-token latency under 50ms, and end-to-end under 5 seconds. Report goodput, not raw success rate, because a 200 that missed every latency target is not a win.

A ramp plan you can run in an afternoon

You do not need a week of testing. You need a deliberate ramp that finds the shape of your degradation curve.

1. Baseline at concurrency 1. Establish clean time to first token, inter-token latency, and cost per request with no contention.
2. Expected peak. Run at your realistic launch-day peak and confirm goodput holds.
3. Two times peak. This is your safety margin. If goodput collapses here, you have no headroom for a good launch day.
4. Push to break. Keep increasing until goodput falls apart or 429s dominate. The breaking point is the number capacity planning actually needs, because it tells you how gracefully the feature degrades rather than just how fast it can go.

Record cost at each step too. A four-hour test at high concurrency can run into thousands of dollars in tokens, so cap it and translate the result into the answer leadership will ask for: what it costs to serve your expected daily active users. If you serve multiple customers on shared limits, this is also where per-tenant cost caps earn their keep, so one heavy tenant cannot starve the rest.

Wire the results into launch, not a slide

The test only pays off if its numbers become guardrails. Turn the 429 ceiling into a concurrency limit and a queue. Turn the goodput target into an SLO your dashboards alert on, which is part of how you monitor an LLM feature in production. Then release behind a flag so you can dial traffic up against the numbers you measured, exactly the kind of staged rollout with a kill switch that lets you back out in seconds if the real curve looks worse than the test.

Frequently asked questions

When should I load test an AI feature?

Before launch, and again on any change that touches the model, prompt size, or provider. Run a quick ramp on every deployment to catch regressions, and a fuller capacity test before a launch or a big traffic event. The earlier you find your rate-limit ceiling, the cheaper it is to design around it.

What tools should I use?

General load tools like k6 or Locust work for driving traffic, but they measure a plain 200 or 500 and miss streaming metrics, so you need to capture time to first token and inter-token latency yourself. Purpose-built LLM load-test tooling records those out of the box. Either way, the corpus of realistic prompts matters more than the tool.

How much does a load test cost?

Real money, because every synthetic request is billed per token. A sustained high-concurrency run can reach four figures. Cap the duration, size the corpus to your real token distribution, and treat the spend as cheap insurance against a failed launch. Track it so you can report cost per thousand requests alongside latency.

What is a good pass mark?

Pick SLOs before you test, then require goodput to hold at two times expected peak. A reasonable starting set is time to first token under 500ms, inter-token latency under 50ms, and end-to-end under 5 seconds, tightened for search-style features and loosened for long-form generation. Set the bar first so the test grades against the product, not the other way around.

A pre-launch load test is a few hours that turns launch day from a gamble into a known quantity. If you would rather hand the whole thing off, standing up a realistic LLM load test and turning its numbers into guardrails is the kind of task Boundev ships in a week.