Context rot: why your AI feature gets worse as the prompt grows

Context rot is the measurable drop in a language model's accuracy as its input context grows, and it starts long before you reach the advertised token limit. A prompt that answers correctly at 4,000 tokens can quietly return wrong answers at 40,000, even on a model that accepts 200,000. The fix is not a bigger context window. It is feeding the model less context, ordered better, with the distractors removed, so the answer it needs is not buried under everything you could have sent.

This matters because most production AI features grow their prompts by accretion. You add a few more retrieved chunks, a longer system prompt, more conversation history, and the eval you ran on a short prompt stops describing what actually ships. Here is what context rot is, how to catch it before your users do, and how to design a feature that stays sharp as its context fills up.

What context rot actually is

Every extra token you put in front of a model is a token it has to read, weigh, and decide is relevant or not. In principle a model with a 200,000-token window treats all of it equally. In practice it does not. As the input gets longer, accuracy on the exact same question degrades, and the decline is not linear. It can stay flat for a while and then fall off a cliff, or drop steadily from the first few thousand tokens, depending on how hard the target answer is to distinguish from the surrounding text.

The practical definition is simple: hold the question and the correct answer fixed, pad the context with more material around it, and watch accuracy fall. That fall is context rot. It is a property of how attention behaves over long inputs, not a bug in any one model, and it shows up across every frontier model we have tested a feature against.

It is not the same as hitting the token limit

The dangerous part is that context rot happens well inside the stated limit. Teams reason that if a model accepts 200,000 tokens, then 40,000 is comfortably safe, so they stop worrying about length. But the token limit tells you when the request will be rejected, not when the answers will get worse. Answers get worse far earlier. Treating the advertised window as a quality guarantee is the mistake that ships a feature which passed every short-prompt test and then fails on the long, real ones.

How context rot shows up in a real feature

Take a support copilot built over a SaaS product's help center, the kind of feature we ship for customers all the time. In development it retrieves the top three passages for a question and answers from them. The eval set is 80 questions, each with a short, clean context, and it scores 94 percent. Everyone signs off and it goes live.

Two weeks later the accuracy on real traffic is closer to 78 percent, and nobody changed the model or the prompt. What changed is the input. Real users ask messier questions, so the retriever now pulls the top ten passages to be safe. Someone added the full account context and the last twenty messages of the conversation to the prompt for personalization. The clean 1,500-token context from the eval is now a 22,000-token context in production. The model is not dumber. It is drowning.

The tell is that the wrong answers are confidently wrong and often quote real text from the context, just the wrong piece of it. That is the signature of context rot rather than a retrieval miss: the right passage is in the prompt, but the model latched onto a plausible neighbor instead. If your feature retrieves the correct chunk and still answers wrong, length is a prime suspect, and it is worth separating from a genuine two-stage retrieval and reranking problem.

Why more context makes accuracy worse

Position matters, and the middle gets lost

Models attend most reliably to the start and the end of their input and least reliably to the middle. Bury the load-bearing passage at token 11,000 of a 22,000-token prompt and the model is measurably less likely to use it than if the same passage sat in the first or last few hundred tokens. This is why the exact same context, reordered, can change the answer.

Distractors compete for attention

Every passage you add that is topically related but does not answer the question is a distractor. The more distractors, the more the model has to discriminate between the real answer and near-misses, and discrimination is exactly what degrades under length. Adding the top ten passages instead of the top three does not add seven helpful passages. It usually adds two more helpful passages and five confident distractors.

Similar-but-wrong beats absent

A context that contains a passage about the annual billing plan when the user asked about monthly is worse than a context with no billing passage at all, because the model will answer from what is present. Long contexts pull in more of these adjacent-but-wrong passages, and each one is a chance for the model to answer the question next to the one that was asked.

How to detect context rot before your users do

The core move is to stop evaluating at one length. Build a length-stratified eval: take the same questions and run them at 2,000, 8,000, 16,000, and 32,000 tokens of context, padding with realistic but irrelevant material from your own corpus. If accuracy holds flat across the rungs, you are fine. If it falls as length rises, you have quantified your context rot and you know the budget at which quality breaks. This is the single most useful test most teams are not running, and it slots directly into the practice of building an eval set from production failures.

A second, cheaper probe is a position test. Take a question you know the model can answer from a single passage, then move that passage to the start, the middle, and the end of an otherwise fixed long context. If the answer flips based on where the passage sits, position sensitivity is live in your feature and reordering will pay off. Track the length of your production prompts as a metric too, next to latency and cost, so you notice when the average context doubles, which it will as your RAG latency budget and retrieval settings drift over time.

How to design around context rot

Retrieve less, rank harder

The instinct to raise top-k until the right chunk is somewhere in the prompt is exactly backwards. A smaller, higher-precision set beats a larger, noisier one because it carries fewer distractors. Spend the effort on ranking quality so that three passages are the right three, rather than sending ten and hoping. Reranking a wide candidate set down to a tight final set is the highest-leverage change here, and it is why trimming context rather than output tends to improve accuracy and cost at the same time.

Put the load-bearing context at the edges

If you know which passage or instruction matters most, place it near the top or the bottom of the prompt, not in the middle. For retrieved passages, order by relevance and put the strongest at the end, closest to the question, where recency helps. This costs nothing and recovers accuracy that pure middle-of-context placement gives away.

Cut duplicates and near-duplicates

Retrieval over real corpora returns the same fact three times from three near-identical documents. Deduplicate before you build the prompt. Three copies of one passage do not triple its weight in a useful way, they just crowd out room for a different passage that would have answered a second part of the question.

Compress history instead of appending it

Conversation history is the quietest source of context bloat. Appending every turn verbatim means a long session eventually spends most of its context on old messages nobody needs. Summarize older turns into a short running state and keep only the last few messages verbatim. This is the same discipline that keeps agents sharp over long runs, closely related to why an AI agent's memory rots in production.

Give the context window a budget

Decide the maximum context your feature will assemble, well below the model's limit, and enforce it. When retrieval plus history plus system prompt exceeds the budget, drop the lowest-value material rather than letting the prompt grow unbounded. A fixed budget turns context from a thing that silently drifts into a thing you control, and it forces the ranking and compression work that actually protects accuracy.

Questions teams ask about context rot

Is a larger context window the real fix?

No. A larger window raises the ceiling on how much you can send, not the quality of answers at a given length. Bigger windows can even make context rot worse in practice, because they remove the pressure that kept prompts short and tempt teams to stuff everything in. The window is a capacity limit, not a quality setting.

Does context rot affect agents, not just RAG?

Yes, and often more. Agents accumulate tool outputs, intermediate reasoning, and history across many steps, so their context grows turn over turn without anyone deciding to make it longer. An agent that was accurate on step three can be unreliable on step fifteen purely because its context tripled in the meantime. The same fixes apply: budget the context, summarize old steps, and keep the current task at the edges of the prompt. Deciding between stuffing everything in and retrieving on demand is the crux of the long context versus RAG decision.

How do I measure context rot on my own feature?

Run your existing eval questions at several context lengths with realistic padding and plot accuracy against length. The length where the line starts to fall is your rot threshold, and it becomes the context budget you enforce in production. If you do not have an eval set yet, that is the first thing to build, before any prompt tuning.

Context rot is not a reason to distrust long-context models. It is a reason to treat context length as a variable you measure and control rather than a free resource. The teams whose AI features stay accurate in production are the ones who send the model the least context that still answers the question, and who test at the lengths their users actually generate.