llm-d

SandboxAI Infrastructure

Open source · CNCF · CNCF Sandbox · Apache-2.0

llm-d is a Kubernetes-native framework for running distributed LLM inference at scale.

updated 2026-07-01

What it is

llm-d is a Kubernetes-native framework for running distributed LLM inference at scale. It sits on top of vLLM and the Kubernetes Gateway API Inference Extension, adding the pieces vLLM doesn't handle on its own: intelligent request scheduling across a fleet of model servers, prefix-cache-aware routing (so requests with shared context land on the replica that already has that KV cache warm), prefill/decode disaggregation (splitting the compute-bound and memory-bound phases of inference onto different hardware), hierarchical KV cache offloading (GPU → CPU → remote storage), and autoscaling driven by actual traffic and hardware utilization rather than generic CPU/memory metrics. It runs across NVIDIA, AMD, Intel, and Google TPU accelerators, which matters if you're not locked into a single vendor.

In short: it's the orchestration layer for teams running vLLM in production at a scale where naive round-robin load balancing and single-node serving stop working.

Who builds it and why

llm-d is a CNCF Sandbox project, built by a coalition that includes Red Hat, Google, IBM, and NVIDIA, alongside the vLLM maintainer community. The motivating problem is real: as organizations move from "call an API" to "run our own inference fleet," they hit the same wall — GPU utilization is poor, KV cache is wasted across requests that could share it, and Kubernetes' default scheduling primitives have no concept of prefix affinity or prefill/decode separation. This is an attempt to solve that at the platform layer instead of every company reinventing it in-house.

158 contributors and steady commit activity (latest confirmed commit 2026-07-01) suggest active, multi-vendor investment rather than a single company's side project.

Production readiness signal

CNCF Sandbox status means it's early-stage by CNCF's own maturity model — sandbox is the entry tier, not incubating or graduated. That's an honest signal: this is not a project with years of battle-testing or a large enterprise adoption track record yet.

That said, 3,613 GitHub stars, 158 contributors, and a v0.8.1 release with active commits point to real engineering investment and growing community traction. The backing from Red Hat, Google, IBM, and NVIDIA is a stronger production signal than star count alone — these are organizations with production GPU fleets who have skin in the game.

Read the versioning literally: pre-1.0 means APIs and CRDs can still shift. Budget for upgrade friction.

Who should use this

Platform teams running vLLM across multiple GPU/TPU nodes who are hitting KV cache and routing inefficiency at the naive load-balancer level.
Organizations serving high-volume, shared-context workloads (RAG, chat with long system prompts) where prefix-cache-aware routing has a measurable cost/latency payoff.
Teams already invested in Kubernetes as the control plane and comfortable operating Gateway API infrastructure.
Multi-hardware shops (mixing NVIDIA, AMD, Intel, TPU) who need a common scheduling layer instead of separate stacks per vendor.

Who should NOT use this

Anyone serving a single model on a single node or small fixed cluster — this is solving a distributed-systems problem you don't have yet. Plain vLLM or TGI is simpler and sufficient.
Teams not running Kubernetes. There's no non-K8s deployment path; adopting llm-d to get its benefits is not a reason to adopt Kubernetes itself.
Risk-averse production environments that require CNCF Incubating/Graduated maturity or long-term API stability guarantees — this is Sandbox, pre-1.0, and will break things.
Teams wanting a fully managed inference product — this is infrastructure you operate yourself, not a hosted service.

Alternatives

vLLM alone — sufficient if you don't need multi-node scheduling, disaggregation, or cross-request cache sharing.
Ray Serve / Ray + vLLM — a more general distributed-compute framework that can serve LLMs but lacks llm-d's inference-specific scheduling primitives out of the box.
NVIDIA Triton Inference Server / Dynamo — vendor-backed alternative with strong NVIDIA hardware integration, less Kubernetes-native and less multi-vendor by design.

Pricing

Fully open source, Apache-2.0 licensed. No paid tier, no vendor lock-in on the license itself — cost is entirely your own infrastructure and operational overhead.

Frequently asked

What is llm-d?+

llm-d is a Kubernetes-native framework for running distributed LLM inference at scale.

Who builds llm-d?+

llm-d is a CNCF Sandbox project, built by a coalition that includes Red Hat, Google, IBM, and NVIDIA, alongside the vLLM maintainer community.

Is llm-d production ready?+

Who should use llm-d?+

Platform teams running vLLM across multiple GPU/TPU nodes who are hitting KV cache and routing inefficiency at the naive load-balancer level. Organizations serving high-volume, shared-context workloads (RAG, chat with long system prompts) where prefix-cache-aware routing has a measurable cost/latency payoff.

Who should not use llm-d?+

Anyone serving a single model on a single node or small fixed cluster — this is solving a distributed-systems problem you don't have yet. Plain vLLM or TGI is simpler and sufficient. Teams not running Kubernetes.

What are the alternatives to llm-d?+

vLLM alone — sufficient if you don't need multi-node scheduling, disaggregation, or cross-request cache sharing. Ray Serve / Ray + vLLM — a more general distributed-compute framework that can serve LLMs but lacks llm-d's inference-specific scheduling primitives out of the box.

How much does llm-d cost?+

Fully open source, Apache-2.0 licensed. No paid tier, no vendor lock-in on the license itself — cost is entirely your own infrastructure and operational overhead.