llmaz

What it is

llmaz (pronounced "lamas") is a Kubernetes-native inference platform for serving large language models. It wraps the operational complexity of running LLM inference—model loading, backend selection, autoscaling, multi-node serving—into a set of CRDs (Model, Playground, Service) that let you declare what you want served and let the operator handle the rest. It supports multiple inference backends (vLLM, SGLang, llama.cpp, and others) under one abstraction, and can pull model weights from Hugging Face, ModelScope, or S3-compatible object storage. It also supports heterogeneous GPU/accelerator scheduling and multi-host inference for models too large for a single node.

Who builds it and why

llmaz is a Go project with 20 contributors and 306 GitHub stars — a small-to-mid-sized community, not a large foundation-backed project. CNCF maturity status is not publicly available, meaning it hasn't been submitted (or hasn't been accepted) into the CNCF Sandbox/Incubation pipeline as of this writing. It's built by practitioners solving a real, narrow problem: Kubernetes doesn't have an opinionated way to run LLM inference workloads, and every team ends up hand-rolling Deployments, HPA configs, and backend glue code to do it. llmaz packages that pattern. The commit cadence (latest activity into 2026) suggests active maintenance, but this is not a project with corporate backing disclosed publicly — treat it as community-driven infrastructure tooling, not a vendor product.

Production readiness signal

Version v0.1.4 is the headline fact here: this is pre-1.0 software. That's not disqualifying, but it means breaking API changes between minor versions are likely, and you should pin versions and read release notes before upgrading. 306 stars and 20 contributors is a modest but real community — enough to indicate the project isn't vaporware, not enough to guarantee fast issue turnaround or long-term maintenance guarantees. There's no publicly available founding date or CNCF maturity tier to anchor trust in institutional backing. Before running this in production, check: open issue count and response times, whether the CRDs have changed shape recently (API stability), and whether the backends you need (vLLM version compatibility, specific accelerator support) are actively tested in CI.

Who should use this

• Platform teams already running Kubernetes who need a standard way to serve multiple LLMs across teams without reinventing scheduling/autoscaling logic each time.
• Teams that need multi-backend flexibility (vLLM today, SGLang or llama.cpp tomorrow) without rewriting deployment manifests.
• Organizations with heterogeneous GPU fleets (mixed accelerator types) who need a scheduler-aware layer instead of hardcoded node affinities.
• Teams comfortable operating pre-1.0 CRDs and willing to contribute upstream or self-support when things break.

Who should NOT use this

• Teams wanting a fully managed inference service — llmaz is infrastructure you operate yourself, not a hosted API.
• Anyone needing rock-solid API stability today; v0.1.4 means the CRD schema can and will change.
• Small teams serving a single model with a simple vLLM Deployment — llmaz adds abstraction overhead you may not need yet.
• Organizations that require CNCF-vetted governance or long-term support contracts as a procurement condition; that signal isn't publicly available here.
• Non-Kubernetes shops — this only makes sense if you're already running K8s.

Alternatives

• KServe — broader ML-serving CRD project under CNCF governance, more mature but heavier and more general-purpose than LLM-specific.
• Ray Serve (with RayLLM/vLLM integration) — better fit if you're already on Ray for distributed compute and want inference as part of a larger pipeline.
• vLLM production stack / raw vLLM + Kubernetes Deployment — simplest option if you only need single-backend serving without multi-backend abstraction.

Pricing

Fully open source, Apache-2.0 licensed. No paid tier, no commercial entity behind it publicly disclosed. You pay only in engineering time to deploy, operate, and upgrade it yourself.