An AI API gateway is the layer between your application and the model providers you call. It gives you one API surface across many models, and in exchange it handles routing, fallbacks, retries, caching, rate limits, spend attribution, logging and content filtering.
Most buying guides list these products in one long table as if they were interchangeable. They are not. Roughly half are self-hosted proxies you run, half are SaaS services that sit in your data path, some are resellers that sell you inference, and a few are endpoints bolted onto a cloud contract. That distinction decides your data-residency posture, your monthly bill and how much you have to operate yourself.
The uncomfortable part, which almost nobody says plainly: a gateway usually adds a layer of cost and latency rather than removing it. The things that genuinely cut your LLM bill are provider-side. Prompt caching, batch APIs, cheaper-model routing. A gateway’s job is to make those measurable and enforceable, not to invent savings on your behalf.
Which kind of AI API gateway do you actually need?
Start by answering one question: does your prompt traffic have to stay inside your own network?
If yes, you want a self-hosted proxy. LiteLLM, Bifrost, Agent Router (formerly Envoy AI Gateway) and the open-source Portkey gateway all run in your VPC or cluster.
If no, and you want per-team budgets plus zero operational burden, you want a SaaS gateway. Cloudflare AI Gateway, Vercel AI Gateway and Helicone all do this. Cloudflare’s core features are free on every plan; Vercel applies zero markup to token prices.
If you want one invoice and one OpenAI-shaped endpoint across hundreds of models without holding provider accounts, you want an aggregator. That is OpenRouter, whose public models API listed 469 models on 17 October 2026.
If you already have a committed cloud agreement, the hyperscaler’s native endpoint is usually the cheapest path to start. Amazon Bedrock, Azure API Management’s AI gateway and Google Vertex AI all do routing, quotas and logging inside an existing contract.
Reading the prices and benchmarks
Prices, API data and repository dates are from 17 October 2026. This is a comparison of published specifications, without hands-on load tests or cache-hit measurements.
The Bifrost benchmark is vendor-run on its own hardware. The RouteLLM cost-savings figures come from a paper first submitted on 26 June 2024. The Not Diamond latency and savings claims are vendor marketing with no published methodology.
What an LLM API gateway actually does
An LLM gateway is a reverse proxy that understands tokens, prompts and model identifiers, rather than just forwarding HTTP. That extra understanding is what separates it from a load balancer.
The jobs are these:
- Unified API surface. One OpenAI-compatible endpoint, so switching providers is a config change rather than a rewrite. TrueFoundry advertises 1000+ models behind one interface; LiteLLM claims 100+ provider integrations.
- Model routing and fallback chains. Send a request to a primary model, then try a second when the first fails or is too slow. Azure API Management’s backend load balancer supports round-robin, weighted, priority-based and session-aware strategies.
- Retries, timeouts and circuit breakers. Bifrost Enterprise trips on provider response headers such as
retry-after-ms. Bifrost says that is deliberate: “no latency heuristics or error-rate windows to tune.” - Caching. Exact-match replay, semantic similarity match, or provider-native prompt caching. Details below, because this is where most buying mistakes happen.
- Rate limiting and quotas. Cloudflare offers fixed and sliding windows. Agent Router’s QuotaPolicy lets you write CEL expressions such as
input_tokens + cached_input_tokens / 10u + output_tokens * 6uso cached tokens cost a tenth of a normal one. - Cost attribution and budget caps. Per key, per team, per user, per project. This is the single most common reason companies adopt a gateway at all.
- Logging and observability. Request metadata, token counts, latency and cost. Agent Router records distributed traces using OpenTelemetry GenAI conventions.
- Prompt and version management. Portkey supports up to three prompt templates on its free tier, unlimited above that. Bifrost Enterprise adds CEL-matched prompt versions with weighted traffic distribution.
- Guardrails and content filtering. Cloudflare evaluates prompts and responses with
@cf/meta/llama-guard-3-8bon Workers AI, so guardrail checks are billed as token inference. Kong ships an AI Sanitizer for PII redaction. - Load balancing across provider accounts. Weighted distribution over several keys for the same provider, so one exhausted key does not take the app down.
The four categories, and why the difference matters
The category a product sits in predicts almost everything about it: who sees your prompts, who you pay, and whether you get an uptime guarantee.
| Category | Examples (verified 17 Oct 2026) | Who operates it | Where your prompts go | Published cost basis |
|---|---|---|---|---|
| Self-hosted open source proxy | LiteLLM, Bifrost, Agent Router, Portkey Gateway, Helicone (Bifrost, Agent Router and Helicone are Apache-2.0; Portkey Gateway is MIT) | You | Your infrastructure | Infrastructure only; Bifrost OSS and Helicone are free |
| SaaS gateway in your data path | Cloudflare AI Gateway, Vercel AI Gateway, Helicone Cloud, Bifrost Enterprise, TrueFoundry | Vendor | Vendor network to provider | Cloudflare: free core, 5% credit fee. Vercel: zero markup, metered features. TrueFoundry: $0 / $25 per user |
| Aggregator or reseller | OpenRouter | Vendor | Vendor network to many providers | No inference markup; fee on credit purchase, plus 5% on BYOK spend above $25,000/month |
| Hyperscaler-native endpoint | Amazon Bedrock, Azure API Management AI gateway, Google Vertex AI | Hyperscaler | Your cloud account | Billed through your existing cloud agreement |
Two categories blur into each other and it matters. Bifrost and Helicone are both open source and sold as hosted enterprise products. Portkey’s enterprise tier is now a Palo Alto Networks product with its own deployment models. When you read a feature list, the first question is not “does it support semantic caching?” but “who runs the box my traffic goes through?”
Model routing: what a silent fallback actually costs you
Model routing means choosing which model handles each request. The basic version is a fallback chain: try the model you want, and if it errors or times out, try another. The advanced version predicts difficulty and routes accordingly. Three mechanisms are in real production, and they are not equally trustworthy.
Rule-based routing by price or latency is the most transparent. OpenRouter’s documented default is to skip providers that saw significant outages in the last 30 seconds, pick among the stable ones weighted by the inverse square of price, and keep the rest as fallbacks. You can override it with sort: "price", "throughput" or "latency", or filter with max_price. Cloudflare’s Dynamic Routing adds flow-based A/B splits and content-aware conditions.
Learned routing picks a model by classifying the prompt. Cloudflare’s Auto Router, documented 2 October 2026, builds a pool of eligible models, excludes those that cannot handle the request’s inputs or whose spend limits are exhausted, classifies the conversation for difficulty, then scores quality against cost. Amazon Bedrock predicts response quality per model in the same family. Not Diamond explicitly says it is not a gateway, at $0.05 per million tokens routed.
The failure mode nobody sells you: a silent fallback changes the answer’s quality, and the user never finds out. Any fallback changes which model produced the text. Cloudflare makes this observable with a
cf-aig-stepresponse header, where0means the primary model served the request and each increment is another fallback, pluscf-aig-routed-modelandcf-aig-routing-reasonfrom Auto Router. If you cannot see which model answered, you cannot evaluate whether the answer is good.
There is a second trap, and it is a caching trap. Routing decisions that change per request destroy prompt caches. OpenRouter handles this with sticky routing, pinning a conversation to the provider that served its cache write, with sessions expiring after ten minutes of inactivity. Cloudflare’s Auto Router does the same with cf-aig-session-id and says outright that without a session ID, requests in the same conversation may go to different models and cannot reuse the prompt cache. Agent Router makes the same trade-off explicit: it switches models at the start of a turn only when the benefit outweighs the cost of losing the cache.
The comparison table
Everything below was read from the vendor’s own page on 17 October 2026. “Vendor-reported” means a marketing or benchmark claim, not an audited figure.
| Product | Category | Published price (17 Oct 2026) | Self-host open models | Caching | Budget caps | Notable limits stated by the vendor |
|---|---|---|---|---|---|---|
| OpenRouter | Aggregator | No inference markup; fee on credit purchase; 5% on BYOK above $25,000/mo list-price cost; $5 min / $25,000 max per transaction | No, provider-hosted only | Provider-native prompt caching + sticky routing | Yes; BYOK spend excluded by default | Credits may expire after 365 days; self-serve DPA with SCCs |
| Vercel AI Gateway | SaaS | Zero markup incl. BYOK; $5/mo free credit; provider allowlist $0.10/1,000 requests; team-wide ZDR $0.10/1,000; traces $0.05/1,000 + $0.50/GB | No | Yes (feature) | Yes, but soft caps; HTTP 402 | Budget check runs at request start, so overshoot is possible |
| Cloudflare AI Gateway | SaaS | Core features free; 5% fee on Unified Billing credits; DLP free; guardrails billed as Workers AI tokens | No | Exact-match only; TTL 60s to 1 month; max request 25 MB | Yes, up to 20 rules/gateway, eventually consistent | DLP buffers streamed responses, raising time-to-first-token |
| Helicone | OSS + SaaS | Free 10k req/mo; Pro $79/mo; Team $799/mo; Enterprise custom | Yes (Apache-2.0) | Cloudflare Workers KV, 300+ edge locations, default 7-day TTL, max 365 days | Via gateway | Hobby ingestion capped at 10 logs/min |
| Portkey / Prisma AIRS | OSS + SaaS | Dev free 10k req/mo; Pro $49/mo for 100k, overage $9 per 100k up to 3M | Yes | Simple + semantic, unlimited TTL | Yes | Fully air-gapped deployment is now legacy and no longer offered |
| TrueFoundry | SaaS + self-host | Developer $0 (10k req/user/mo); Pro $25/user/mo (20k included, $20 per extra 100k); Enterprise custom | Yes | Yes | Yes; 1 rule on Developer, 10 on Pro | Retries inside the gateway do not count toward usage |
| Bifrost | OSS + enterprise | OSS free forever (Apache-2.0); Enterprise custom, 14-day trial | Yes | Direct (hash) + semantic; default 0.8 cosine threshold; skips conversations over 3 messages | Yes (Enterprise) | Vendor benchmark says 11 µs overhead at 5k RPS; the GitHub description says under 200 µs |
| Agent Router (ex-Envoy AI Gateway) | OSS + hosted | OSS free (Apache-2.0); Tetrate hosted and enterprise tiers | Yes, incl. Kubernetes InferencePool | Provider-agnostic cache_control; min 1,024 tokens; max 4 breakpoints | QuotaPolicy per model, CEL cost expressions | Quota windows cap at 1 day; service-wide quota exposed in API but not enforced |
| LiteLLM | OSS + enterprise | OSS free; Enterprise custom | Yes | Redis, Valkey, Qdrant semantic, S3, GCS, disk, in-memory | Yes, virtual keys with max_budget | In-memory cache is not shared across workers; hit rate drops roughly by worker count |
| Kong AI Gateway | Enterprise platform | Plus from $25/mo, 1M requests included, $200 per extra 1M, 10M/mo max; model proxy $100/mo per model | Yes | AI Semantic Cache + AI Prompt Compressor | Yes; SSO and audit logs are Enterprise only | Plus is capped at 10M requests/month |
| Azure API Management | Hyperscaler | AI gateway tier listed as “pricing details are coming soon”; v2 tiers include 10M / 50M / unlimited API requests | Yes, self-hosted endpoints | Semantic caching, requires Azure Managed Redis or RediSearch-compatible store | Yes, token rate limiting and quotas | Anthropic Messages API only in v2 tiers; unified model API is preview |
| Amazon Bedrock | Hyperscaler | Billed on your AWS agreement; batch inference 50% lower; simple prompt optimiser $0.03 per 1,000 tokens | Yes, on Bedrock | Provider prompt caching | Yes, via Bedrock quotas and inference profiles | Intelligent prompt routing works only within one model family |
| Not Diamond | Learned router (not a gateway) | $0.05 per million tokens routed | Router-agnostic | No | No | Vendor-reported 100-150 ms added latency; no published methodology |
| RouteLLM | OSS router (stale) | Free (Apache-2.0) | Yes, via LiteLLM | No | No | Last commit 10 August 2024; paper first submitted 26 June 2024 |
What a gateway actually costs you
Two vendors say they add nothing to your token bill. Both add something else.
OpenRouter’s FAQ states plainly that it “passes through the pricing of the underlying providers without any markup.” Its Terms of Service, last updated 31 August 2026, are more precise: fees include “payment processing fees and transaction fees,” with a $5 minimum and $25,000 maximum per transaction, and OpenRouter reserves the right to expire unused credits 365 days after purchase. The number that bites at volume is bring-your-own-key: per the BYOK documentation, the pay-as-you-go plan includes $25,000 per month of list-price inference cost with no BYOK fee, and usage above that is charged a 5% fee.
Vercel states that AI Gateway “adds zero markup to provider token prices, including with BYOK.” Then the pricing page itemises the extras: custom reporting at $0.075 per 1,000 tag writes and $5 per 1,000 queries; a team-wide provider allowlist and team-wide zero data retention at $0.10 per 1,000 requests each; and trace drains at $0.05 per 1,000 traces plus $0.50 per GB of egress. Pro plans include no allowance for the trace meters, so charges begin with the first delivered trace.
Cloudflare is the most generous on paper: caching, rate limiting, analytics and DLP scanning are free on all plans. Use Unified Billing with Cloudflare-managed credentials and you pay a 5% fee on credits, and ZDR applies. Use your own keys instead and there is no credit fee, but ZDR does not apply. Either way, guardrails are billed as Workers AI token inference.
Kong bills per million requests. Plus starts at $25 per month with 1 million requests included, $200 per additional million, and a 10 million monthly ceiling. The AI Gateway Manager add-on is $100 per month per proxied model, capped at five models on Plus. Advanced analytics run at $20 per 1M requests above the first million, metering is $20 per million ingested events, and billing integration is 0.4% of billing volume up to $250,000.
TrueFoundry is the clearest example of a gateway priced on its own layer. Pro is $25 per user per month with 20,000 requests per user pooled across the team, and $20 per additional 100,000. Their own estimator shows seven users and one million gateway requests landing at $355 per month in gateway fees, before any model-provider cost at all.
The honest summary: a gateway bills you for the control plane, and the control plane is not free. Only Bifrost OSS and Helicone’s self-hosted stack are genuinely $0, and both leave the infrastructure bill with you.
What genuinely lowers your LLM bill
Since the gateway is a cost centre, the levers that matter are the ones underneath it. These are all provider-side and all verifiable.
Prompt caching is the biggest one. Anthropic’s pricing is explicit: a 5-minute cache write costs 1.25x base input price, a 1-hour write costs 2x, and cache reads cost 0.1x base input price. Some models are cheaper still, at 0.025x on Claude Fable 5.1 and Claude Mythos 5.1, and 0.05x on Claude Opus 5.5 and Claude Sonnet 5.5. Minimum cacheable prompt length runs from 512 to 4,096 tokens depending on model, with a 20-block lookback window for finding a prior write.
Batch APIs halve the bill for anything that does not need an answer now. OpenAI’s Batch API offers a 50% discount with a 24-hour completion window, up to 50,000 requests per batch, a 200 MB input file and up to 2,000 batches per hour. Anthropic’s Message Batches API charges 50% of standard prices, caps batches at 100,000 requests or 256 MB, and expires anything unfinished after 24 hours.
Smaller-model routing pays when the router is right. Amazon Bedrock states that intelligent prompt routing “can reduce costs by up to 30 percent without compromising on accuracy.” Not Diamond claims 20-40% for its customers. Both are vendor figures and the second has no published methodology.
Budget caps stop the tail, which is where the real money leaks. A single runaway coding agent can burn a month’s budget overnight; Palo Alto Networks’ Prisma AIRS launch post makes exactly that argument.
Caching: exact match, semantic match, and why the numbers are missing
Exact-match caching stores a response under a hash of the request and replays it only when the hash matches. Cloudflare AI Gateway documents the default precisely: the cache key is a SHA-256 hash of the provider, endpoint, model, provider authentication header, and the full request body. Any difference in messages, tools or model parameters is a separate entry. TTL ranges from a minimum of 60 seconds to a maximum of one month, and hits and misses are reported in the cf-aig-cache-status header.
Semantic caching embeds the prompt and serves the closest stored answer above a similarity threshold. It catches rephrased questions that exact-match never will. It is also the fastest way to serve one user’s answer to another user. Bifrost’s defaults encode that worry: a 0.8 cosine similarity threshold, a conversation-history threshold of 3 that skips caching entirely for longer conversations, cache_tool_call_responses off by default because “a replayed tool call runs in a later caller’s agent loop with arguments chosen for someone else’s prompt,” and identity scoping on by default.
Now the part worth internalising. Almost no gateway publishes a real semantic-cache hit rate. Cloudflare says caching reduces latency “by up to 90%” and Bifrost ships benchmarks, but neither publishes a customer hit-rate distribution. Anthropic’s published figures show how traffic-dependent this is:
“Users typically experience cache hit rates ranging from 30% to 98%, depending on their traffic patterns.” : Anthropic, on prompt caching inside Message Batches
There is a privacy problem too, and it is not subtle. If your prompt embeds a user ID, an account number or a session, a semantic cache will happily serve user A’s answer to user B. Bifrost’s answer is to require a cache key and scope entries by identity. Helicone lets you namespace with a cache seed and ignore specific JSON keys, at the cost of fewer hits. Cloudflare warns that cache hits skip DLP scanning, so a response cached under a permissive policy keeps being served after you tighten that policy until the TTL expires. If you cache personalised or regulated content, self-host and scope by identity, or do not cache it at all.
How to roll one out: seven steps
1. Establish the baseline before you buy anything. Pull thirty days of provider invoices and per-model token counts. You cannot tell whether a gateway worked if you do not know what you spent before.
2. Decide the category, not the brand. Prompts must stay in your network? Self-hosted proxy. One invoice across many models? Aggregator. Already committed to a cloud? That hyperscaler. The AI Unpacker tool directory and reviews section are reasonable starting points for the shorter evaluations.
3. Start in observation mode. Route 100% of traffic through the gateway but keep your existing client behaviour. Agent Router has a shadow mode that evaluates quota rules without ever rejecting; Cloudflare has analytics before you enable anything else. Log every request’s model, tokens, latency and cost.
4. Turn on fallbacks deliberately, with observability. Enable the fallback chain, then assert that the response tells you which model answered. If your gateway cannot say, fix that before shipping, because a silent downgrade is worse than an outage you can see.
5. Add provider-native prompt caching first. It is the largest saving and it does not change answer behaviour. Use Anthropic’s cache pre-warming with max_tokens: 0 to load your system prompt before users arrive.
6. Only then add gateway-level response caching. Start exact-match with a short TTL. Add semantic caching only for a specific, non-personalised endpoint where you have measured the hit rate yourself.
7. Set budget caps last, in shadow mode, then enforce. Watch the telemetry, confirm the number, then turn enforcement on. Expect a soft cap: Vercel’s own docs say spend “can end up slightly over the budget.”
What breaks: failure modes and gotchas
Fallback changes the answer. The single biggest operational surprise. A user gets a worse model’s output and blames your product. Assert the serving model in your own telemetry and alert on fallback rate.
Semantic caching breaks agentic traffic. LiteLLM puts it bluntly: semantic caching “suits single-shot prompts and goes badly wrong on agentic traffic.” Every tool result changes the prompt. Bifrost simply refuses to cache past three messages.
In-memory caches silently underperform. LiteLLM documents that an in-memory cache lives inside one worker process, so a proxy running four workers keeps four separate caches and “the hit rate drops roughly by the worker count.” You will not see an error. You will just save less than you expected.
Spend attribution can be spoofed. LiteLLM warns that end users can pass a user field in the request body, shifting reported spend away from the key owner. The fix exists (overwrite_user_with_key_hash, from v1.95.0) but is off by default.
BYOK spend may not count toward your budget. OpenRouter states that BYOK inference spend does not count toward guardrail or workspace budgets by default, which means “a budget limit can appear far from its cap even after significant BYOK usage.” Vercel says the same.
Guardrails cost money and latency. Cloudflare bills guardrail checks as Workers AI token inference. Kong’s semantic guards add a model call to your request path. Budget for them separately.
Quota enforcement can fail open. Agent Router’s QuotaPolicy needs Redis and a dedicated rate-limit service that its own Helm chart does not deploy today, and controller.quotaRateLimitFailureModeDeny defaults to false, so requests continue unenforced if that service is unreachable. Quota windows cap at one day; monthly and yearly windows need Envoy Gateway’s separate usage-based rate limiting. The API also exposes a service-wide quota that is not yet enforced.
DLP breaks streaming. Cloudflare documents that with DLP response scanning on, streamed responses are fully buffered before being scanned, so time-to-first-token rises proportionally to the entire generation time. Setting the DLP policy to check requests only avoids this.
Security, governance and the third party in your data path
Key management. Every serious gateway stores provider keys rather than embedding them in applications. Cloudflare offers BYOK with encrypted storage and rotation for 20+ providers, with a documented credential precedence: a provider key on the request wins, then a stored BYOK key, then Unified Billing. Bifrost Enterprise integrates with HashiCorp Vault and the AWS, GCP and Azure secret managers. LiteLLM supports scheduled key rotation with a grace period.
Zero Data Retention is the control most buyers ask about, and the definitions differ. Vercel’s AI Gateway “uses zero data retention by default” and permanently deletes prompts and responses after requests complete. Cloudflare’s ZDR applies only to Unified Billing requests using Cloudflare-managed credentials, not to BYOK. OpenRouter exposes a per-request zdr flag that restricts routing to endpoints with a zero-retention policy, and if you pin a provider manually that does not match your privacy setting, the request errors rather than completing.
Certifications are not the same as a DPA. Portkey lists SOC 2, ISO 27001, GDPR and HIPAA for its enterprise deployments. LiteLLM Enterprise is SOC 2 Type 2 and ISO 27001 with self-hosted or air-gapped deployment. Helicone is SOC-2 Type II and GDPR compliant and Apache-2.0 licensed. Not Diamond is SOC-2 and ISO 27001 certified. None of that tells you whether your prompts pass through a third party in production.
Data residency is the real question. OpenRouter offers EU in-region routing via eu.openrouter.ai, where requests are decrypted in the EU and routed only to shared provider endpoints in Europe, covering the EU/EEA, UK and Switzerland. It states plainly that account and billing data may be stored or processed in the US, and that self-serve business has no contract, with the DPA including standard contractual clauses. If that is not enough, Portkey’s hybrid deployment puts the gateway and data plane in your environment with only the control plane hosted, or you run Bifrost, LiteLLM or Agent Router yourself.
PII handling before the prompt leaves. Kong’s AI Sanitizer redacts PII before it reaches the upstream provider. Cloudflare’s DLP scans both prompts and responses and supports flag or block actions. Both add latency and cost.
Legal review. The following is general information. Controller and processor relationships, DPA terms, SCC coverage and residency obligations under GDPR depend on your specific processing. If a gateway will hold personal data about EU or UK residents, get your counsel to read the vendor’s DPA before choosing a gateway.
The bottom line
Buy the category, not the feature checklist. Self-host when prompts cannot leave your network. Buy managed when you would rather not run Postgres, Redis and a rate-limit service. Use an aggregator when you want one invoice across hundreds of models. Start with your hyperscaler if you already have the contract.
Then be honest about the economics. A gateway adds a control plane and a bill. The savings live one layer down, in prompt caching at 0.1x input price, in batch APIs at 50% off, in routing easy prompts to cheap models, and in the budget cap that stops the agent that decided to loop.
If you take one thing from this, make it visibility. A gateway you cannot audit, whose served model you cannot read from the response, and whose cache you cannot scope by identity will cost you more in debugging than it ever saved on inference.
FAQ
What is an AI API gateway in one sentence?
It is a proxy that sits between your application and one or more LLM providers and exposes a single API, usually OpenAI-compatible, so one codebase can call many models while the gateway handles routing, fallbacks, retries, caching, rate limits, spend tracking, logs and guardrails. Cloudflare, Vercel, LiteLLM, Kong and OpenRouter all do some subset of that, which is why comparing them as if they were interchangeable wastes weeks.
Does an AI API gateway reduce my LLM bill?
Usually not on its own. A gateway normally adds a platform fee: Cloudflare charges a 5% fee on credits bought through Unified Billing, Kong bills $200 per additional million AI gateway requests on its Plus plan, and OpenRouter charges a fee plus payment processing on credit purchases. What genuinely lowers spend is provider-side prompt caching, batch APIs at 50% off, routing simple prompts to cheaper models, and budget caps that stop runaway agents.
Which AI API gateway is best for a small team with no data residency needs?
Vercel AI Gateway or Cloudflare AI Gateway. Vercel applies zero markup to token prices and includes $5 of monthly credit per team, billing you only if you enable features like team-wide zero data retention or custom reporting. Cloudflare’s core features, including caching, rate limiting, analytics and DLP scanning, are free on all plans, and you pay the 5% credit fee only if you use Unified Billing with Cloudflare-managed keys.
Should I self-host LiteLLM or buy a managed gateway?
Self-host LiteLLM if your prompts cannot transit a third party or you need custom routing logic your vendor will not build. Buy a managed one if you would rather not operate Postgres, Redis and a rate-limit service. LiteLLM Enterprise is self-hosted with SOC 2 Type 2 and ISO 27001 controls, 24/7 support with a one-hour Sev 0 response target, and air-gapped deployment, but the price is custom.
What is the difference between exact-match and semantic caching for LLMs?
Exact-match caching replays a stored response only when the request hashes identically, so any change to the body is a miss. Cloudflare AI Gateway does this by default, hashing the provider, endpoint, model, auth header and full request body with SHA-256. Semantic caching embeds the prompt and serves the nearest stored answer above a similarity threshold, which catches rephrased questions but can serve one user’s answer to another. Bifrost defaults to a 0.8 cosine threshold and refuses to cache conversations longer than three messages.
What hit rate should I expect from LLM caching?
Almost nobody publishes a defensible number, so assume you will have to measure it. The clearest published figure is Anthropic’s, which says users typically see batch prompt-cache hit rates ranging from 30% to 98% depending on traffic patterns. For gateway-level response caching the risk is structural: exact-match caches miss whenever anything in the body changes, and LiteLLM notes that an in-memory cache is not shared between workers, so hit rate drops roughly by the worker count.
Does my prompt data leave my network when I use a SaaS AI gateway?
Yes, it transits the vendor, which makes them a sub-processor under your contracts. OpenRouter states it does zero logging of prompts and completions by default and offers a 1% discount if you opt in to logging, but its privacy policy still collects inputs containing personal data, and its Terms reserve the right to expire unused credits after 365 days. Vercel deletes prompts and responses after each request by default. If that is unacceptable, self-host LiteLLM, Bifrost or Agent Router, or use Portkey’s hybrid deployment, which puts the gateway in your VPC.
Are per-team budget caps hard or soft limits?
Mostly soft. Vercel documents its AI Gateway budgets as soft caps, because the check runs at the start of a request, so the request that crosses the limit still completes and total spend can end up slightly over. Cloudflare’s spend limits are described as eventually consistent for the same reason, a burst of concurrent requests can briefly exceed the limit. Both return an error to the client rather than silently truncating output.
Is OpenRouter just a reseller with a markup?
It is an aggregator, and on inference price it claims no markup at all. Its FAQ says it passes through provider pricing without markup but charges a fee when you purchase credits, and its Terms allow payment processing and transaction fees, with a $5 minimum and $25,000 maximum per transaction. The number to watch is bring-your-own-key: on the pay-as-you-go plan the first $25,000 per month of list-price inference cost carries no BYOK fee, and anything above that is charged a 5% fee.
What happened to Envoy AI Gateway and Portkey?
Both were absorbed. Envoy AI Gateway became Agent Router and joined the Agentic AI Foundation on 10 September 2026, with the same code, maintainers and APIs, and the GitHub repository moved from envoyproxy/ai-gateway to theagentrouter/agent-router. Palo Alto Networks closed its acquisition of Portkey and shipped Prisma AIRS AI Gateway as generally available in July 2026. Any comparison you find that still lists those two names is out of date.
Sources
- OpenRouter FAQ: no-markup claim, credit-purchase fee, BYOK allowance of $25,000 per month and 5% fee above it, EU routing, zero-logging default and 1% opt-in discount
- OpenRouter Terms of Service: last updated 31 August 2026; payment processing and transaction fees, $5 minimum and $25,000 maximum per transaction, 365-day credit expiry
- OpenRouter Privacy Policy: last updated 31 August 2026; collection of inputs containing personal data, SCCs, no use of inputs or outputs for model training
- OpenRouter BYOK documentation: 5% BYOK fee, prioritized versus fallback keys, ZDR and data-region key declarations, BYOK spend excluded from budgets by default
- OpenRouter provider routing: default inverse-square-of-price load balancing, 30-second outage window, sort and max_price options, rolling 5-minute percentile metrics
- OpenRouter prompt caching: per-provider cache write and read multipliers, sticky routing with 10-minute session expiry
- OpenRouter models API: 469 models, six
openrouter/*router slugs, 78 batch variants, 16 free variants (queried 17 October 2026) - Cloudflare AI Gateway pricing and Unified Billing: 5% credit fee, free DLP, guardrails billed as Workers AI inference
- Cloudflare AI Gateway caching: SHA-256 cache key over provider, endpoint, model, auth header and request body; 60-second to 1-month TTL;
cf-aig-cache-status - Cloudflare AI Gateway limits: 25 MB cacheable request, 10 free and 20 paid gateways, 200 requests per 60 seconds on Unified Billing
- Cloudflare Auto Router: last updated 2 October 2026; classification and scoring, default model pool, session affinity headers, routing-reason values
- Cloudflare fallbacks:
cf-aig-stepheader semantics - Cloudflare spend limits: 20 rules per gateway, eventual consistency, 429 on breach, best-effort cost estimation
- Cloudflare AI Gateway DLP: last updated 12 October 2026; streaming buffering and TTFT impact, cache hits skip DLP
- Vercel AI Gateway pricing: zero markup, $5 monthly credit, custom reporting, allowlist and ZDR rates, trace drain meters
- Vercel AI Gateway budgets: soft-cap semantics, 402
quota_for_entity_exceeded, refresh periods, BYOK excluded - Vercel AI Gateway security and compliance: ZDR by default, provider and model allowlists
- Anthropic prompt caching: 1.25x and 2x write multipliers, 0.1x and discounted read multipliers, 20-block lookback, minimum cacheable prompt lengths
- Anthropic Message Batches API: 50% pricing, 100,000 request and 256 MB caps, 30-98% cache hit rate range
- OpenAI Batch API: 50% discount, 24-hour window, 50,000 requests and 200 MB per batch, 2,000 batches per hour
- AWS Bedrock pricing: intelligent prompt routing up to 30% cost reduction, 50% batch discount, $0.03 per 1,000 tokens for the simple prompt optimiser
- AWS Bedrock intelligent prompt routing: supported model families, English-only limitation, fallback-model criteria
- Azure API Management AI gateway capabilities: token rate limiting, semantic caching requirements, load balancer strategies, circuit breaker behaviour
- Azure API Management pricing: AI Gateway tier pricing not yet published; v2 tier request allowances
- TrueFoundry pricing: Developer and Pro tiers, $20 per additional 100,000 requests, gateway fee example, model costs pass-through
- TrueFoundry AI Gateway documentation: provider coverage, supported APIs, deployment options
- Portkey feature comparison: last modified 1 October 2026; Dev 10k and Pro $49/month with $9 per 100k overage, retention periods, deployment models
- Portkey open-source gateway: MIT licensed, 13,203 stars, last pushed 25 May 2026, 1,600+ models, latency and footprint claims, retry and guardrail capabilities
- Palo Alto Networks Prisma AIRS AI Gateway GA announcement: acquisition of Portkey, 68 trillion tokens claim, MCP traffic growth figures
- Helicone pricing: free, $79, $799 and Enterprise tiers; ingestion limits, retention windows, certifications
- Helicone caching documentation: Cloudflare Workers KV storage, default and maximum TTL, cache buckets and seeds
- Helicone open-source repository: Apache-2.0, self-hosting options, 100+ model gateway
- Bifrost pricing: free OSS tier and Enterprise features
- Bifrost semantic caching: 0.8 cosine threshold, 5-minute TTL, three-message conversation limit, identity scoping, tool-call policy
- Bifrost benchmarks: 5,000 RPS, 11 µs overhead on t3.xlarge, 59 µs on t3.medium, exclusions from the overhead figure
- Bifrost GitHub repository: Apache-2.0, 8,867 stars, last pushed 17 October 2026, and the conflicting “under 200µs” overhead claim
- Agent Router GitHub repository: Apache-2.0, 2,236 stars, last pushed 16 October 2026, verified via the GitHub API on 17 October 2026
- Agent Router documentation: rename from Envoy AI Gateway, Agentic AI Foundation membership on 10 September 2026
- Agent Router QuotaPolicy: CEL cost expressions, shadow mode, one-day maximum window, unenforced service-wide quota
- Agent Router prompt caching: 1,024-token minimum, four-breakpoint maximum, provider translation
- LiteLLM caching documentation: cache backends, worker-local in-memory caveat, agentic-traffic warning on semantic caching
- LiteLLM spend tracking: user-supplied identity warning, 10,000-row count cap on spend logs
- LiteLLM Enterprise: SOC 2 Type 2, ISO 27001, SSO and SCIM, air-gapped deployment, SLA response targets
- Kong AI Gateway documentation: AI Semantic Cache, AI Prompt Compressor, AI Sanitizer, guardrail integrations
- Kong pricing: Plus plan from $25 per month, $200 per additional million requests, $100 per model proxy, metering rates
- Not Diamond pricing: $0.05 per million tokens routed, 20-40% savings claim, 100-150 ms latency claim, SOC-2 and ISO 27001
- Not Diamond security documentation: updated 15 June 2026; privacy-preserving routing, TLS and at-rest encryption
- RouteLLM repository: Apache-2.0, router types, last commit 10 August 2024 (verified via GitHub API on 17 October 2026)
- RouteLLM paper, arXiv:2406.18665: submitted 26 June 2024, revised 23 February 2025; up to 85% cost reduction claim
- GitHub REST API: repository pushed_at timestamps for RouteLLM, LiteLLM, Bifrost and the Envoy AI Gateway rename used throughout
