Building a Multi-Provider LLM Gateway: What I Learned Cloning OpenRouter
August 10, 2026
Notes on designing a provider-abstraction layer that lets an app switch LLM providers with zero client-side changes, plus what actually made routing latency predictable.
Every app that calls an LLM eventually hits the same wall: you've hard-coded calls to one provider's SDK, and now you want to add a second one, or fall back to it when the first is rate-limited. Building a gateway that sits in front of multiple LLM providers behind a single API was mostly an exercise in hiding that complexity from everything downstream.
The provider abstraction layer
The core idea is a thin interface every provider adapter implements: given a normalized request (model, messages, streaming flag), return a normalized response or stream. Each adapter translates that into whatever shape the underlying provider actually expects, and translates the response back. Nothing above that layer — the API routes, the client SDK, the billing/usage tracking — needs to know which provider is actually handling a given request.
That abstraction is what makes zero-downtime provider switching possible. If a provider starts erroring or a model gets deprecated, you swap the adapter registration, not the API contract. Clients never see a breaking change.
Getting streaming right
Response streaming was the part that took the most iteration. Different providers chunk tokens differently, and some wrap chunks in provider-specific metadata you don't want leaking through your API. The gateway normalizes every provider's stream into the same server-sent-events shape before it reaches the client, so a frontend only ever has to handle one streaming format regardless of which model answered.
Latency budget matters here too — every extra hop of buffering or transformation adds delay you can measure. Keeping the routing and normalization logic lean (no unnecessary JSON parsing round-trips, no blocking I/O in the hot path) is what kept sub-200ms routing latency achievable even under concurrent load.
Designing the SDK developers actually want
A gateway is only as good as the SDK wrapped around it. The goal for the TypeScript SDK was that switching from calling a provider directly to calling the gateway should be close to a one-line change — same method names, same response shape, with the provider selection as just another parameter. Cutting that integration boilerplate down is what actually reduces onboarding time; a technically elegant abstraction that still requires rewriting call sites isn't a win.
Where this breaks down
The honest limitation: full normalization is easy for chat completions and gets harder fast for provider-specific features — function calling schemas, vision inputs, and reasoning-token formats don't map cleanly onto each other. The gateway exposes an escape hatch (a raw passthrough mode) for those cases rather than pretending every provider is interchangeable, because pretending that would just push the incompatibility further downstream where it's harder to debug.