LLM provider lock-in: design your product for a model swap

Model neutrality means your product can switch LLM providers with a config change instead of a rewrite. You get it by putting one thin seam between your application and any provider SDK, normalizing the four things providers actually disagree on (tool calling, structured output, system-prompt handling, and token accounting), and eval-gating every swap. It is not free: you trade a provider's sharpest features for the freedom to leave, so build the seam only where a swap is plausible.

Most teams do not decide to lock themselves to one provider. They just call the SDK directly from forty places, shape their prompts around that model's quirks, and discover the lock-in the first time they need to leave and cannot. This is a design problem you can get ahead of, and it costs far less to solve on day one than during an outage.

Why a model swap stops being optional

Early on, one provider is a reasonable bet. Then the reasons to move start arriving, and they rarely arrive on your schedule:

• A provider has a multi-hour degradation and your feature is down with it, because every call points at one endpoint with no alternative to fail over to.
• A cheaper or stronger model ships mid-quarter and your competitor adopts it in a week while you scope a rewrite.
• Your provider deprecates the exact model version your prompts were tuned against, with a fixed sunset date.
• An enterprise deal requires data residency or a specific vendor, and the answer to "can you run this on another model?" decides the contract.
• You want pricing leverage, and you have none if both sides know you cannot actually leave.

Each of these is survivable if a swap is a configuration change. Each is a fire drill if the provider is woven through your code. Neutrality is not about distrusting your current vendor. It is about keeping the option to move cheap, the same way you keep failover cheap for a database you have no plans to replace.

What actually leaks between providers

"Just wrap the API" undersells the problem. The HTTP call is the easy part. The lock-in lives in the four places providers quietly disagree, and a naive wrapper leaks all of them into your product.

Tool calling schemas

Every provider supports function or tool calling, and no two describe it the same way. The JSON shape of a tool definition differs, the field that carries the model's chosen call differs, parallel tool calls are handled differently, and the format for returning a tool result back to the model differs. If your agent code reads those provider-specific shapes directly, every tool call site is a lock-in point.

Structured output

Getting reliable JSON out of a model is one of the most provider-specific surfaces there is. One provider has a strict schema-constrained mode, another leans on a JSON flag plus a prompt instruction, a third does best with a tool call whose arguments are your schema. Behavior on an unparseable response also differs. Build your feature around one provider's strongest mode and you have hardwired that provider. Our own guidance on getting structured outputs reliable in production is worth reading before you standardize on any single mechanism.

System prompts and roles

Providers treat the system role differently: some weight it heavily, some fold it into the first user turn, some cap its length or ignore parts of it. A prompt that lands cleanly on one model can drift on another with no code error at all, only worse answers.

Token accounting and limits

Tokenizers differ, so the same text is a different token count on each provider. Context limits, max-output caps, and rate-limit units (requests versus tokens per minute) all vary. If your budgeting, truncation, or cost-attribution logic assumes one tokenizer, it silently miscounts the moment you switch.

Where to put the seam

The fix is one thin adapter layer that your application talks to, and that talks to each provider SDK. Your product code should never import a provider SDK directly.

Give the adapter a small, provider-neutral contract: a request carries messages, tools, a response schema, and generation settings in your own vocabulary; a response carries the text, any tool calls, a finish reason, and normalized token usage. Each provider gets one adapter that translates in and out of that contract, and that adapter is the only file that knows a provider's tool-call shape or structured-output quirks. Keep prompts as data, not as strings assembled with provider-specific control tokens, so the same prompt can be rendered for any adapter.

In practice the seam usually lives inside an internal service rather than a library scattered across your app. If you already route calls through an internal AI gateway, that is the natural home for the adapter, because it already owns retries, budgets, and cost attribution. Neutrality is what makes real provider failover possible: the resilience patterns that keep an AI feature up only work if there is a second provider to fail over to, and routing each request to the cheapest capable model assumes you can address more than one.

What not to abstract

Neutrality taken too far becomes its own trap. If you flatten every provider to a lowest common denominator, you give up the very features you are paying for, and your feature gets worse on all of them.

Let the adapter expose provider strengths behind a capability flag instead of hiding them. If one provider has a strict structured-output mode that removes a whole class of parsing failures, use it when that adapter is active and fall back to a prompt-plus-validate path when it is not. The neutral contract is the request and response shape, not a promise that every model behaves identically.

And do not build the seam for a model you will never leave. A single low-stakes call to one provider, wrapped behind an interface you invented in an afternoon, is enough. The heavy version earns its keep only where a swap is genuinely plausible: your highest-volume calls, anything on the critical path, and anything an enterprise contract might force onto another vendor.

The hidden cost of neutrality

Being honest about the price keeps this from becoming architecture for its own sake. A real neutrality layer costs you in four ways. You maintain one adapter per provider, and each is a place bugs hide. You need an eval per provider, because a prompt tuned on one model is not validated on another until you measure it. You sometimes accept the lowest-common-denominator feature to keep the contract clean. And prompts drift: the wording that scored well on your incumbent may underperform elsewhere, so a swap is never purely mechanical.

The way to pay that cost safely is to treat a provider swap exactly like a model upgrade: gate it on evals. The same discipline that catches a regression when your provider updates a model catches a regression when you move providers. Run your labeled eval set against the new adapter, compare against the incumbent, and only then flip the flag. Nothing here should reach users on the strength of a demo.

A worked example

Take a B2B support copilot, built fast against one provider. Tool calls are read straight from that provider's response shape at roughly forty call sites, the JSON extraction depends on its schema-constrained mode, and token budgeting uses its tokenizer. It works well for months.

Then the provider has a six-hour degradation during a customer's business day. The team has an account with a second provider, but there is no way to point traffic at it: the tool-call parsing, the structured-output path, and the token math are all specific to the first vendor. The copilot is simply down until the incident clears.

Afterward they build the seam. One adapter interface, two implementations, prompts moved into a neutral template, token usage normalized to a single internal unit. They stand up a labeled eval set of a few hundred real support tickets and run it against both adapters, so "is the second provider good enough" is a number rather than a hope. The refactor touches a lot of files once, but now a provider swap is a flag flip validated by evals, and they ramp it the same careful way they would ship any AI change without breaking production, behind a flag with an instant rollback. The numbers here are illustrative, but the shape is the point: the expensive work was undoing the lock-in, not the swap itself.

When you do not need this yet

If you are prototyping, or you have a single call site, or a contract genuinely fixes you to one vendor for the foreseeable future, skip the machinery and wrap the call behind a thin interface so you are not painted into a corner. The goal is not maximum abstraction. It is to make sure that on the day a swap becomes worth it, the answer is a configuration change and an eval run, not a quarter of rework. For the broader set of decisions around building AI features that survive contact with production, our AI engineering playbook collects the patterns that tend to matter, and the case for owning a small model for a narrow high-volume call is the far end of the same spectrum.

Common questions

Is model neutrality the same as multi-model routing?

No. Routing decides which model handles each request to optimize cost or quality. Neutrality is the lower layer that makes any of those models addressable through one contract. You need the seam before routing across providers is even possible, though routing within a single provider does not require it.

Does an AI gateway give me neutrality for free?

A gateway is the right place to put the seam, but it does not grant neutrality on its own. If the gateway passes provider-specific tool-call and structured-output shapes straight through, the lock-in just moves to whoever calls the gateway. Neutrality comes from normalizing those shapes, wherever that logic lives.

Will one prompt work across every provider?

Rarely, on the first try. The prompt is portable, but its performance is not until you measure it. Keep prompts as neutral templates so the same prompt can target any adapter, then validate each provider against your eval set and adjust where needed. Treat prompt performance as provider-specific even when the prompt text is shared.

How many providers should I actually support?

Usually one in production plus one validated fallback is enough to defeat lock-in and enable failover. Supporting more only pays off when routing, pricing leverage, or compliance genuinely requires it, and every extra adapter is ongoing maintenance and another column in your eval matrix.