Envoy-ai-gateway

What it is

Envoy AI Gateway is a control plane and set of Envoy Gateway extensions purpose-built for routing, securing, and observing traffic to generative AI backends — think LLM APIs (OpenAI, Anthropic, Bedrock, Azure OpenAI, self-hosted vLLM/Ollama endpoints) — through a single unified ingress layer. It sits on top of Envoy Gateway (which itself sits on top of Envoy Proxy) and adds AI-specific primitives: provider-agnostic request/response translation, token-based rate limiting, API key virtualization, model routing/failover, and usage metering. Instead of every team wiring up their own SDK-specific client logic and quota tracking per LLM provider, you point traffic at the gateway and it normalizes the mess.

It's a Kubernetes-native project — deployed via CRDs (AIGatewayRoute, AIServiceBackend, etc.) that extend the Gateway API, not a bolt-on sidecar or standalone proxy you have to hand-roll config for.

Who builds it and why

Built under the Envoy Gateway umbrella, with backing and active contribution from Tetrate (a primary Envoy/Envoy Gateway commercial steward) alongside contributors from Bloomberg, Google, and others in the Envoy ecosystem. The motivation is straightforward: as organizations plug in multiple LLM providers and self-hosted models, they hit the same problems API gateways solved for microservices a decade ago — auth sprawl, inconsistent rate limiting, no centralized observability — but for a new traffic shape (token-based cost/quota instead of request-count-based). Rather than build a parallel proxy stack, the project extends Envoy Gateway, which means it inherits Envoy's battle-tested data plane instead of reinventing one.

Production readiness signal

118 contributors and 1,795 GitHub stars is a healthy, actively-worked project for its niche — not a weekend side project, but also not yet at Envoy-proxy-core scale of adoption evidence. It shipped a v1.0.0 release, which signals the maintainers consider core APIs stable enough to commit to compatibility guarantees, a meaningful bar for a CRD-driven project. Commit activity through mid-2026 shows it's still being actively developed rather than coasting. CNCF maturity level is not publicly available, meaning it likely hasn't been submitted or accepted into the CNCF Sandbox/Incubation/Graduated tracks yet — treat it as pre-CNCF-vetted, so do your own bake-off before betting production traffic on it without a fallback plan.

Who should use this

• Platform teams already running Envoy Gateway (or willing to adopt it) who need centralized LLM traffic management instead of per-team, per-provider client wiring.
• Organizations juggling multiple LLM providers/models where consistent rate limiting, failover, and cost attribution across teams matters more than provider-specific SDK features.
• Teams that want Kubernetes-native config (CRDs, Gateway API) rather than a separate proxy product with its own config language and lifecycle.
• Anyone building internal "AI platform as a service" who needs to enforce quotas and virtualize API keys without asking every downstream team to change code.

Who should NOT use this

• Teams not already on Envoy Gateway/Gateway API — adopting this just to get AI gateway features means taking on the whole Envoy Gateway operational surface, which is a real cost if you're happy with NGINX ingress or a simpler proxy.
• Single-provider, single-team shops with light LLM usage — the SDK's native client plus a rate limiter is simpler and has less operational overhead.
• Anyone needing a fully managed, zero-ops solution — this is self-hosted infrastructure you operate, patch, and upgrade yourself.
• Risk-averse production environments requiring CNCF-graduated or long-track-record tooling — v1.0.0 is recent, and the project hasn't accumulated years of production war stories yet.

Alternatives

• Kong AI Gateway — Kong's plugin-based approach to LLM traffic management, mature if you're already on Kong.
• LiteLLM Proxy — lighter-weight, Python-based LLM proxy focused on provider abstraction and cost tracking, less Kubernetes-native.
• Portkey / Cloudflare AI Gateway — hosted/managed alternatives if you'd rather not operate the gateway yourself.

Pricing

Fully open source under Apache-2.0. No paid tier, no license gating — you self-host and operate it. Cost is your own infrastructure and engineering time, not license fees.