● Beta — KubeEra is in active development. Profiles are live; AI-perception measurement is rolling out.
KubeEra
K

Kafka

App Definition & Delivery
Open source · Apache-2.0

Kafka is a distributed event streaming platform: an append-only, partitioned, replicated log that you publish records to and consume records from.

updated 2026-07-01

What it is

Kafka is a distributed event streaming platform: an append-only, partitioned, replicated log that you publish records to and consume records from. It's used as a durable message bus, a system-of-record for event data, and increasingly as a substrate for stream processing (via Kafka Streams or ksqlDB). It's not a traditional message queue — it retains data, supports replay, and scales through partitioning rather than clustering a single queue. Category-wise it sits under App Definition and Development, but in practice it's infrastructure: most teams treat it as a piece of the data plane, not application code.

Who builds it and why

Originally built at LinkedIn to handle high-throughput activity tracking and log aggregation, Kafka was open sourced and became an Apache project. It's now maintained by a broad contributor base (1,704 contributors on GitHub) spanning Confluent, LinkedIn, and independent contributors from companies running it at scale. The motivation hasn't changed much since inception: decouple producers from consumers, handle massive write throughput, and give downstream systems a durable, replayable record of what happened. CNCF maturity level is not publicly available — Kafka is an Apache Software Foundation project, not a CNCF one, which is worth noting if you're checking a CNCF landscape box.

Production readiness signal

33,053 GitHub stars and 1,704 contributors indicate a large, active ecosystem — this isn't a project you have to worry about disappearing. Last commit activity as of 2026-07-01 shows ongoing development. License is Apache-2.0, which is permissive and safe for commercial use without copyleft concerns. Latest release version isn't listed here, so check the Apache Kafka release page directly before you commit to a version for a new deployment — KRaft (removing ZooKeeper dependency) has matured significantly in recent versions and changes your operational model meaningfully if you're starting fresh. Founded date isn't available in this dataset, but Kafka has been in production at extreme scale (LinkedIn, Netflix, Uber, etc.) for over a decade, which is the real signal here more than star count.

Who should use this

Teams with genuine high-throughput event streaming needs: clickstream data, log aggregation, change-data-capture pipelines, microservices event buses, or any system where multiple consumers need independent, replayable access to the same stream of events. If you're building event-driven architectures with several downstream services that each need to process events at their own pace, or you need durable retention with replay for debugging/reprocessing, Kafka is a legitimate fit. Organizations with dedicated platform/SRE capacity to run and tune a distributed log are the right audience — this is not a "deploy and forget" system.

Who should NOT use this

If you need a simple task queue — one producer, one consumer, no need for replay — Kafka is overkill; use SQS, RabbitMQ, or even a Postgres-backed queue. If your team doesn't have the operational bandwidth to manage partitioning, replication, consumer group rebalancing, and JVM tuning, you'll spend more time firefighting Kafka than building on it. If your message volume is modest (a few hundred messages/sec) and you don't need long retention or replay, you're paying complexity tax for capacity you don't need. And if you need strict FIFO ordering across an entire topic without partition-key discipline, Kafka's per-partition ordering guarantee will bite you.

Alternatives

  • Apache Pulsar — similar pub/sub-plus-streaming model with built-in multi-tenancy and tiered storage, decouples compute/storage more natively.
  • AWS Kinesis — managed alternative with less operational overhead, but you trade portability and fine-grained control for convenience.
  • RabbitMQ — better fit for traditional queueing/task distribution patterns where replay and log retention aren't requirements.

Pricing

Kafka itself is fully open source under Apache-2.0 — no licensing cost. Your real cost is operational: compute, storage, and the engineering time to run brokers, monitor consumer lag, manage upgrades, and handle rebalancing. Managed offerings (Confluent Cloud, AWS MSK, Aiven) exist if you want to offload operations, priced separately by the vendor.

Frequently asked

What is Kafka?+
Kafka is a distributed event streaming platform: an append-only, partitioned, replicated log that you publish records to and consume records from. It's used as a durable message bus, a system-of-record for event data, and increasingly as a substrate for stream processing (via Kafka Streams or ksqlDB).
Who builds Kafka?+
Originally built at LinkedIn to handle high-throughput activity tracking and log aggregation, Kafka was open sourced and became an Apache project. It's now maintained by a broad contributor base (1,704 contributors on GitHub) spanning Confluent, LinkedIn, and independent contributors from companies running it at scale.
Is Kafka production ready?+
33,053 GitHub stars and 1,704 contributors indicate a large, active ecosystem — this isn't a project you have to worry about disappearing. Last commit activity as of 2026-07-01 shows ongoing development. License is Apache-2.0, which is permissive and safe for commercial use without copyleft concerns.
Who should use Kafka?+
Teams with genuine high-throughput event streaming needs: clickstream data, log aggregation, change-data-capture pipelines, microservices event buses, or any system where multiple consumers need independent, replayable access to the same stream of events.
Who should not use Kafka?+
If you need a simple task queue — one producer, one consumer, no need for replay — Kafka is overkill; use SQS, RabbitMQ, or even a Postgres-backed queue.
What are the alternatives to Kafka?+
Apache Pulsar — similar pub/sub-plus-streaming model with built-in multi-tenancy and tiered storage, decouples compute/storage more natively. AWS Kinesis — managed alternative with less operational overhead, but you trade portability and fine-grained control for convenience.
How much does Kafka cost?+
Kafka itself is fully open source under Apache-2.0 — no licensing cost. Your real cost is operational: compute, storage, and the engineering time to run brokers, monitor consumer lag, manage upgrades, and handle rebalancing.