What an AI Gateway Actually Does (and Why Platform Teams Keep Building One)

Every company that gets past its first two or three AI features runs into the same problem. One team calls a model provider directly with an API key in an environment variable. Another team uses a different provider through a different SDK. A third team built a wrapper "for now." Six months later nobody can answer simple questions: how much are we spending, which apps send customer data to which vendor, and what happens when a provider has an outage?

That is the moment platform teams start building an AI gateway. The name sounds grander than the thing. It is a proxy with opinions.

The basic shape

An AI gateway is a service that every application calls instead of calling model providers directly. The application sends a request that looks like a normal chat or completion call. The gateway decides which model actually serves it, attaches the real credentials, records what happened, and returns the response.

That one indirection buys you a lot.

Credentials live in one place. Application teams get a gateway token, not a provider key. When a key needs to rotate, you rotate it once. When a team should lose access, you revoke one token.

Usage becomes attributable. Because every request passes through, you can tag it with the calling app, team, and environment. Token counts come back in provider responses, so the gateway can log input and output tokens per request and turn that into cost per team. This is usually the feature finance asks about first.

Routing becomes a config change. If a newer model is cheaper and good enough for summarization, you can move the summarization route without touching application code. If a provider is degraded, the gateway can fail over to a second provider for routes where that is acceptable.

Policy has a home. Rate limits, allowed models per app, maximum context sizes, and rules about which data classifications can go to which vendor all need to be enforced somewhere. Enforcing them in each application means enforcing them inconsistently.

Logging is the hard part

Logging sounds like the easy win, and it is where most of the real design work hides.

Prompts and responses often contain exactly the data you are trying to protect. If you log every prompt in full, your gateway's log store becomes the most sensitive database in the company. If you log nothing, you cannot debug, audit, or evaluate anything.

The approach I prefer is to separate metadata from content. Metadata (who called, which model, token counts, latency, status, cost) is always logged and kept a long time. Content is logged only for routes that opt in, is redacted or classified before storage, and has a shorter retention period with tighter access. Make that split explicit early, because retrofitting it after an audit question is painful.

Things a gateway should not do

It is tempting to keep adding features until the gateway becomes an AI platform in its own right. A few things usually belong elsewhere.

Prompt templates and business logic. If the gateway starts rewriting prompts for specific apps, it becomes a shared dependency that every product team needs to change. Keep prompts with the applications that own them.

Heavy content filtering in the hot path. Some screening belongs in the gateway, like blocking obvious secrets or regulated identifiers from leaving the network. Long-running classification on every request adds latency to every call. Push expensive checks to asynchronous review where you can.

Being the only safety control. A gateway cannot know whether an agent should be allowed to send an email or delete a record. Those decisions belong in the application and its tool permissions, not in a proxy that only sees text going in and out.

Build, buy, or borrow

You do not have to write one from scratch. Open-source proxies exist that speak a common, OpenAI-compatible request format and translate it for other providers. Cloud providers and API management vendors also sell gateway features. Whatever you choose, check three things: whether it supports streaming responses cleanly, whether you control where logs go, and whether you can add your own policy checks without forking it.

When it is worth it

If you have one application and one provider, a gateway is overhead. Use the provider SDK and move on.

Once you have several teams, more than one provider, or any compliance requirement about where data goes, the gateway stops being optional. The question is only whether you build it on purpose or end up with five accidental ones scattered across your codebase.

The best sign that it is working is boring: application teams stop thinking about it. They call one endpoint, they get a response, and the questions about cost, access, and data flow finally have answers.