Cut LLM input-token cost by trimming context, not output

Most teams optimizing LLM cost start with the output: shorter responses, cheaper models, tighter max_tokens. That is worth doing, and we covered it in controlling output token cost. But on most production features, the output is not where the money goes. The input is. A chat feature that sends the full conversation history on every turn, a RAG endpoint that stuffs ten chunks into the prompt, or a system prompt that has grown to two pages, all pay for those tokens on every single call. Input tokens are cheaper per unit than output, but there are often five to ten times more of them, and they grow silently as the product does.

• Input context is usually the larger half of the bill. Measure the input/output split before you optimize anything.
• The three biggest sources of input bloat are conversation history, over-retrieved RAG context, and an oversized system prompt with stale few-shot examples.
• Trimming input often improves answer quality too, because the model stops drowning in irrelevant context.

Where input tokens actually come from

Before cutting anything, look at one real request and count where the tokens are. In most features the breakdown is some mix of four things: the system prompt and instructions, few-shot examples, retrieved context (for RAG), and conversation history. One or two of those usually dominate. A support copilot's cost is almost all retrieved documents and history; a classification endpoint's cost is almost all system prompt and examples. You cannot trim effectively until you know which one you are fighting, and the answer differs per feature. Tools that do per-feature cost attribution make this split visible instead of guessed.

The reason this matters: input bloat compounds. Output length is bounded by what the model decides to say. Input length is bounded only by whatever you keep appending, and the default in most codebases is to append and never remove. A 20-turn conversation can carry 5,000 to 10,000 tokens of history when the model realistically needs 500 to 1,000 tokens of recent context to answer the next turn well. You pay that difference on every turn for the rest of the session.

Trim conversation history

The simplest lever is a sliding window: keep the last N turns verbatim and drop the rest. It is crude but it caps the growth, and for many chat features the last few turns carry almost all the relevant context anyway. The failure mode is losing a fact stated 15 turns ago that still matters, so a plain window is the right default only when older turns rarely carry forward.

When they do carry forward, summarize instead of dropping. A condenser replaces old turns with a running summary once the history crosses a threshold, so the thread stays coherent without carrying every raw message. Published agent systems using a summarizing condenser have cut API cost by roughly 2x on long conversations with no measurable drop in task quality. The trick is to compact proactively, on a rolling basis, rather than waiting until you are about to overflow the context window. By the time you hit the limit you have already paid full price for a hundred bloated turns.

Right-size RAG context

RAG features have a specific and expensive habit: retrieving a fixed, generous number of chunks and pasting all of them into the prompt, whether or not the question needs them. Ten chunks feels safer than five, so ten it is, on every query. That is often the single largest line in the input budget.

Two changes help. First, retrieve fewer but better chunks. Better retrieval quality, for example the reranking in hybrid search with BM25 and embeddings, lets you drop from ten chunks to three or four without losing the answer, because the top chunks are actually the right ones. Second, make the count dynamic: an easy, specific question needs one chunk; a broad question needs more. A lightweight relevance filter that trims the retrieved set to what actually scores well can shrink the per-step input dramatically. One published approach used a small model to filter context down from 10,000 tokens to 500 to 1,000 per step, and total cost dropped 57 percent even after accounting for occasional re-requests.

Prune the system prompt and few-shot examples

System prompts accumulate. Every incident adds a sentence, every edge case adds an example, and nobody deletes anything because deleting feels risky. Over a year a prompt doubles, and you pay for the whole thing on every call. Audit it: many instructions are redundant, contradictory, or no longer earning their tokens.

Few-shot examples are the most expensive part and often the most removable. They exist to steer behavior you could not get from instructions alone. Once you have an eval suite, you can test removing examples one at a time and keep only the ones that measurably change the output. Many features carry five or six examples when two do the real work. If the stable part of your prompt is large and cannot shrink further, prompt caching that prefix is the next lever, since a cached prefix bills at a steep discount on reuse.

Measure before and after, per feature

Every one of these changes trades tokens against quality, so none of them is safe to ship blind. Log input and output token counts per request, split by feature, and watch both cost and an answer-quality metric as you trim. The goal is not the smallest possible prompt; it is the smallest prompt that holds quality steady. Sometimes trimming even raises quality, because a model given only relevant context stops getting distracted by the rest. But you only know which way it went if you measured, and the split that dominates one feature may be irrelevant in the next.

Sequence it by size. Start with whatever the attribution data says is largest, usually conversation history or RAG context, because that is where a 30 percent cut moves the bill the most. The system prompt and few-shot pruning come next. Caching a stable prefix is the finishing move once the variable parts are already lean.

Frequently asked questions

Are input tokens really more expensive than output overall?

Per token, output usually costs more. But in total, input often dominates because there is far more of it: long system prompts, retrieved documents, and accumulated history all bill on every call. Measure the actual split for your feature before assuming either way; it varies a lot between a short-prompt classifier and a long-context copilot.

Will trimming context hurt answer quality?

It can, which is why you measure. In practice, removing irrelevant context often improves answers, because the model is not diluted by noise. The risk is dropping something that mattered, so use summarization rather than hard truncation when older context can still be relevant, and gate every change behind an eval.

How is this different from prompt caching?

Caching reuses a stable prefix so you are billed less for sending the same tokens again; it does not make the prompt shorter. Context trimming makes the prompt genuinely smaller by sending less. They are complementary: trim the variable parts (history, retrieved chunks), then cache whatever stable prefix remains.

What is proactive compaction?

Compacting or summarizing the running context on a rolling basis, before you approach the context window limit, rather than only when you are about to overflow it. Waiting until the limit means you have already paid full price for every bloated turn along the way; compacting early keeps the input lean throughout the session.