What to monitor after you ship an LLM feature to production

The day an LLM feature goes live is the day your test coverage stops being the thing that catches problems. Real users send inputs your evals never imagined, traffic spikes hit concurrency you never load-tested, and the model provider ships a silent update that shifts behavior. A dashboard that answers one question, is this feature healthy right now, is what turns a live AI feature from a liability into something you can run.

This is the monitoring setup we put behind an LLM feature once it is in front of customers. It is deliberately short. A monitoring plan you cannot read in ten seconds is one nobody looks at until the incident is already an hour old.

The four signals that actually matter

Standard application monitoring covers uptime and HTTP errors, and you still need that. But an LLM feature can be up, returning 200s, and quietly producing garbage or burning your margin. Four signal groups catch the failure modes that generic monitoring misses.

Latency, at the tail

Track p50, p95, and p99, plus time-to-first-token if you stream. The average will lie to you: a feature with a 2-second median can still have a p95 of 12 seconds that makes a fifth of your users think it hung. Tail latency is what people actually feel. If you stream responses, time-to-first-token matters more than total time, because it is what tells the user the feature is working. We covered why in streaming LLM responses as a SaaS feature.

Cost, per request and per tenant

Tokens drive both cost and latency, so a token spike is usually the first sign of a bug: a retrieval that grew too large, a prompt-injection loop, a retry storm. Track tokens per minute and cost per hour against a daily budget, and break cost down per tenant. One account running 40x the token volume of everyone else is either your best customer or your next incident. If you run a shared model across tenants, cap them; we go through how in per-tenant LLM cost caps.

Refusal and error rate

Separate infrastructure errors (timeouts, 500s, provider outages) from model refusals (the model declined or returned an empty or invalid answer). They have different causes and different fixes. A climbing refusal rate often means your prompt is colliding with real inputs it was never tested against, and no HTTP monitor will ever show it because those requests succeed at the transport layer.

User feedback

A thumbs-up and thumbs-down on each output is the cheapest quality signal you will ever get, and the only one that reflects whether users actually find the feature useful. Log the feedback linked to the exact request that produced it, so a thumbs-down is a click away from the input, the retrieved context, and the output. Rising thumbs-down while your operational metrics stay green is the clearest early sign of quality drift.

Design the dashboard for the on-call engineer

The test for your dashboard is simple: can someone who did not build the feature open it during an incident and know within ten seconds whether it is healthy? If the answer is no, it is a data dump, not a dashboard.

Put current status in the top row: error rate, p95 latency, cost per hour versus budget, and refusal rate, each with a green, yellow, or red state against a threshold you set in advance. Below that, the same metrics as time series so you can see whether a number is a blip or a trend. Everything else, per-tenant breakdowns, token histograms, individual traces, lives on a second screen you open only when the top row tells you something is wrong.

Set the thresholds before launch, not during the first incident. Decide what p95 is acceptable, what cost per request is too high, what refusal rate warrants a page. Thresholds you argue about while the graph is red are thresholds you set too late.

Trace every request end to end

Metrics tell you something is wrong. Traces tell you what. For an LLM feature, a trace is the full record of one request: the user input, the retrieved context, the exact prompt sent, the model and version, the raw output, the token counts, and the latency of each step. Without it, debugging a bad output is guesswork.

The moment this pays off is when a user reports a wrong answer. With tracing, you pull the request, see that retrieval returned the wrong document, and fix the retrieval. Without it, you are trying to reproduce a non-deterministic output from a screenshot. Sample traces at 100 percent early in the rollout when volume is low, then sample down as traffic grows and storage cost matters.

Tracing and evals are related but not the same thing, and teams often conflate them. Tracing tells you what happened on a specific request in production; evals tell you whether the feature meets a quality bar across a fixed dataset. You want both, and we drew the line between them in observability versus evals.

Watch for silent provider drift

The failure mode people forget is the one they do not control: the model provider changes behavior under a version alias you pinned loosely, or deprecates the model you built on. Your code did not change, your tests still pass, and the outputs shifted anyway. Pin exact model versions, log the version on every request, and keep a small set of reference inputs you run on a schedule so a behavior change shows up as a metric rather than a support ticket. Managing which prompt and model version is live is its own discipline; we covered it in prompt versioning for production AI features.

Frequently asked questions

What is the single most important metric to monitor?

If you can only watch one thing, watch cost per request against a budget, because a runaway token spike is both the most common bug and the one that costs real money by the minute. But cost alone will not catch quality drift, which is why user feedback is the essential second signal. One protects your margin, the other protects the product.

Do I need a dedicated LLM observability tool?

Not to start. You can log tokens, latency, model version, and feedback to the same system you already use for application metrics, and get most of the value. A dedicated tool earns its place once you want automated quality scoring on production traces and per-request replay, but a homegrown dashboard on your existing stack is a fine first version.

How is monitoring different from the evals I run before shipping?

Evals run before release against a fixed dataset and answer whether the feature is good enough to ship. Monitoring runs continuously against live traffic and answers whether it is still healthy right now. Evals catch regressions you can predict; monitoring catches the ones you cannot, like a traffic spike or a provider change. You need both, and neither replaces the other.

How long should I keep traces?

Keep full traces for as long as you realistically debug back, often 30 to 90 days, and keep the aggregated metrics far longer since they are cheap. Sample heavily early in a rollout when volume is low and every trace teaches you something, then reduce the sampling rate as traffic grows and full-fidelity storage stops being worth the cost.