MODULE 01 OF 12 · RELEASE 1

Model Gateways & Provider Abstraction

One front desk for every AI service you use, so swapping one out does not mean rewriting your app.

So what is this really about?

Your app should talk to AI services the same way. Rather than every part of your code knowing how to phone a particular AI company — their exact format, their quirks, their error messages — you build one "reception desk" that everything goes through. Behind that desk you can switch companies, add a backup, or handle a service being busy, and the rest of your app never notices.

The words you will hear

Provider

The company whose AI you are using. There are several, and they all speak slightly differently.

Gateway

Your reception desk: the one piece of code allowed to talk to providers directly.

Abstraction

Hiding the messy differences behind one simple way of asking, so callers do not care who answers.

Rate limit

A provider saying "you are asking too fast, slow down" — your desk handles that, your app does not see it.

By the end you will be able to

• Describe why talking to a provider from everywhere makes change expensive.
• Build one interface that hides provider differences.
• Handle a busy or broken provider inside the gateway, not in your features.

Going deeper

Why this is the first module

Almost every AI project starts the same way. Someone finds the provider’s quickstart, pastes four lines into the file they happen to be working in, and it works. That is genuinely the right way to start — you learn more in ten minutes than in an hour of planning.

The trouble arrives quietly, three months later. By then those four lines have been copied into eleven files. Each copy has drifted: one has a retry, one has a timeout, two have different model names, and one has a hard-coded API key that someone meant to move. Now the provider raises prices, or has a bad week, or releases a model that is twice as good — and the change you want to make is spread across eleven places, one of which you will forget.

This module is first because it is the cheapest problem to fix early and one of the most expensive to fix late.

What "one front desk" actually means

The idea is small: exactly one module in your codebase is allowed to talk to a provider. Everything else asks that module. Nothing else imports the provider’s library — not "usually", not "except for this one script". One place.

That single rule buys you several things at once, and it is worth being precise about what they are:

• Change in one place. A new model name, a new provider, a different timeout — one file.
• Consistent behaviour. Every call gets the same retry policy and the same timeout, because there is only one implementation of them.
• One error type. Your features handle "the AI call failed", not six different provider exception classes.
• Measurement for free. Every call passes one chokepoint, so counting tokens and timing is one piece of code rather than eleven.

That last one matters more than people expect. You cannot control a cost you cannot attribute, and attributing it later means finding all eleven call sites again.

Designing the request

Your gateway needs its own idea of what a request is — not the provider’s. Start smaller than feels right:

askTheModel({
  messages,        // the conversation so far
  expecting,       // the shape you want back
  budget,          // how long, and how much, you will allow
})

Notice what is missing. No temperature, no top-p, no provider-specific knobs. You can add them later if a real need appears, and most never do. Every option you expose is an option every caller has to understand and every provider has to support, so the smallest interface that does the job is the one to start with.

budget deserves a word. It is tempting to leave timeouts to a global default. Do not: a background job summarising documents can reasonably wait thirty seconds, and a request with a person watching cannot. The caller knows which it is, so the caller says.

Designing the response

The result your gateway returns should be equally plain, and it should include the things you will want later even though you do not need them yet:

{
  ok: true,
  value,           // what the model produced
  usage,           // tokens in, tokens out
  tookMs,          // how long it really took
  provider,        // who answered, for when you need to know
}

And when it fails, a result rather than an explosion:

{
  ok: false,
  reason: 'timeout' | 'rate_limited' | 'refused' | 'invalid_output' | 'unavailable',
  retryable: true,
}

Two things are doing real work here. First, failure is a value your caller can inspect, not an exception thrown from three layers down that some code forgot to catch. Second, reason is your vocabulary. Providers describe failure inconsistently and change their wording without warning; the whole point of the desk is that your features never see that.

Doing the extraction safely

Moving working code is where people break things, so the order matters.

1. Write the interface first, with one implementation that does exactly what today’s code does. No retries, no timeouts, no improvements. Just a move.
2. Change the call sites, one at a time, running the tests after each. Resist tidying anything else while you are in there.
3. Only once every call site is moved, start improving the inside.

The reason for that discipline: if you improve behaviour during the move, then something breaks, you cannot tell whether the move or the improvement caused it. Keeping them separate means every failure has one obvious suspect.

Retries, and how they go wrong

Once the extraction is done, retries are the first thing to add inside. They are also the easiest thing to get subtly wrong, in three ways.

Retrying things that should not be retried. If the provider says your request was malformed, sending it again unchanged will produce the same rejection. Retry what is genuinely transient — being rate-limited, a timeout, a server error — and fail fast on everything else.

Retrying immediately. A provider that just told you to slow down will tell you again. Wait, and wait longer each time: a second, then two, then four.

Retrying in lockstep. This one bites hard and is invisible in testing. If a provider has a brief wobble and a thousand of your requests all fail at the same moment, they all wait exactly one second and then all retry at the same moment — a thundering herd that keeps the provider down. The fix is a small random amount added to each wait, so the crowd spreads out. It is called jitter, it is three lines, and skipping it turns a two-second blip into a five-minute outage.

Adding the second provider

You do not need a second provider on day one. You do need the second implementation, and you should write it before you believe your interface is right — because writing it is how you find out.

Almost every leak shows up here. A field you named after the first provider’s terminology. An assumption that a particular parameter exists. A response shape that only one of them produces. These are cheap to fix at this point and expensive once features depend on them.

The test to aim for: run your whole suite against provider one, switch a single configuration value, run it again, and watch the same assertions pass. If that works, your features genuinely do not know who is answering, which was the goal.

The rule that keeps it true

Abstractions decay. Someone in a hurry imports the provider directly "just for this one thing", and six months later you are back where you started without noticing.

So enforce it mechanically. A test that walks your source files and fails if anything outside the gateway imports a provider library is about fifteen lines, and it is the difference between a rule you agreed to and a rule that holds. Write it while the codebase is clean — it is much harder to introduce once there are already violations to argue about.

What you should notice afterwards

When this module is done well, two things become easy that used to be hard. Trying a new model is a configuration change rather than a project. And answering "what did last month cost, broken down by feature?" is a query rather than a guess.

Neither is dramatic. Both are the difference between a system you can steer and one you can only add to.

How it is put together

A single gateway module sits between application code and every provider SDK. Callers pass a neutral request — messages, a schema, a budget — and receive a neutral result. Provider-specific translation, retries, timeouts and token accounting live inside. Nothing outside the gateway imports a provider SDK, which is the property that makes provider choice reversible.

Where you start

A project with one working provider call written directly into a route handler — the shape most codebases actually start in. The lab is the extraction.

The lab

1. Define the neutral request and response types, including a structured error rather than throwing raw provider exceptions.
2. Move the existing call behind the interface without changing behaviour, proving it with the tests that already pass.
3. Add a second provider implementation and a configuration switch.
4. Add a per-request timeout and a bounded retry with backoff, inside the gateway.
5. Record tokens and latency per call so cost is attributable later.

What goes wrong, on purpose

The scenario. The primary provider returns 429 for a sustained period, then begins returning 500s.

What should happen. The gateway retries within its bound, surfaces a structured exhaustion error, and never blocks a request thread indefinitely. Calling code handles one error type regardless of which provider failed.

How you prove it works

• A test double simulating 429 then success proves the retry path.
• A test asserting no file outside the gateway imports a provider SDK.
• A test swapping providers by configuration with identical assertions passing for both.

What you walk away with

A gateway module with two provider implementations, a passing suite that does not know which provider is configured, and per-call token and latency figures in your logs.