● Beta — KubeEra is in active development. Profiles are live; AI-perception measurement is rolling out.
KubeEra
A

Apache Spark

App Definition & Delivery
Open source · Apache-2.0

Apache Spark is a distributed compute engine for large-scale data processing — batch, streaming, SQL, ML, and graph workloads, all on the same runtime.

updated 2026-07-01

What it is

Apache Spark is a distributed compute engine for large-scale data processing — batch, streaming, SQL, ML, and graph workloads, all on the same runtime. It replaced the MapReduce-era assumption that every stage of a pipeline had to hit disk, using in-memory DAG execution instead. Core abstractions are RDDs (low-level, rarely touched directly anymore) and DataFrames/Datasets (the practical interface most teams use). It runs on YARN, Kubernetes, Mesos, or standalone, and reads/writes basically everything: Parquet, ORC, JDBC, Kafka, object stores, Delta/Iceberg/Hudi table formats.

If you've built a data platform in the last decade, you've touched Spark — directly, or through something wrapping it (Databricks, EMR, Dataproc, Synapse).

Who builds it and why

Apache Spark is an Apache Software Foundation project, originally out of UC Berkeley's AMPLab. It's not a CNCF project — maturity classification here is not publicly available because it doesn't apply the same way it does to CNCF-hosted tools.

The contributor base is large and corporate-backed: 3,493 contributors on GitHub is not a hobby-project number. Databricks (founded by Spark's original authors) drives a significant share of core development because their commercial product is built on it, but contributions also come from the Hadoop-ecosystem vendor world (Cloudera, AWS, Microsoft, Google) and independent data engineers fixing things that break at scale. The incentive alignment is straightforward: cloud vendors sell managed Spark, so they fund the engine.

Production readiness signal

43,531 GitHub stars and continuous commit activity through mid-2026 indicate an actively maintained project, not something coasting on past adoption. Founding date and latest release version are not publicly available in this dataset, but Spark has shipped major versions (1.x through 3.x, with 4.x work ongoing) on a predictable cadence for over a decade — this is not an early-stage tool by any measure.

Written in Scala, running on the JVM, Apache-2.0 licensed — no licensing surprises, no relicensing risk like some post-2020 data infra projects.

The real signal is operational: Spark runs the batch and streaming layer for a large fraction of Fortune 500 data platforms. Failure modes are well documented, tuning knobs are well understood (executor memory, shuffle partitions, skew handling), and the on-call playbooks exist because thousands of teams have already hit the same walls.

Who should use this

  • Teams running batch ETL at a scale where single-node tools (Pandas, DuckDB) fall over — think multi-TB to PB daily volumes.
  • Teams needing one engine across batch, streaming (Structured Streaming), and ML (MLlib) instead of stitching three separate systems together.
  • Platform teams already on Kubernetes or YARN who want a mature scheduler integration rather than a newer, less-proven engine.
  • Anyone already inside the Databricks/EMR/Dataproc ecosystem — Spark is the path of least resistance there.

Who should NOT use this

  • Small-data teams. If your datasets fit on one machine, Spark's overhead (JVM startup, cluster coordination, shuffle costs) will cost you more in ops time than it saves in compute. Use DuckDB or Polars instead.
  • Low-latency streaming. Spark Structured Streaming has micro-batch latency floors in the hundreds of milliseconds to seconds. If you need true sub-100ms event processing, look at Flink.
  • Teams without dedicated data infra ownership. Spark tuning (partitioning, memory, skew) is a real skill; under-resourced teams often end up fighting the engine instead of shipping pipelines.
  • Simple SQL-only workloads on modern cloud warehouses. If Snowflake or BigQuery already does the job, adding Spark is unnecessary complexity.

Alternatives

  • Apache Flink — better fit for true low-latency, event-at-a-time streaming.
  • Trino/Presto — better for interactive federated SQL queries, not built for heavy batch ETL or ML.
  • DuckDB — better for single-node analytical workloads where a cluster is overkill.

Pricing

Fully open source, Apache-2.0. No cost to run Spark itself. Real spend comes from infrastructure (compute clusters) and, optionally, managed platforms (Databricks, EMR, Dataproc, Synapse) that charge for orchestration, autoscaling, and support on top of the open-source engine.

Frequently asked

What is Apache Spark?+
Apache Spark is a distributed compute engine for large-scale data processing — batch, streaming, SQL, ML, and graph workloads, all on the same runtime. It replaced the MapReduce-era assumption that every stage of a pipeline had to hit disk, using in-memory DAG execution instead.
Who builds Apache Spark?+
Apache Spark is an Apache Software Foundation project, originally out of UC Berkeley's AMPLab. It's not a CNCF project — maturity classification here is not publicly available because it doesn't apply the same way it does to CNCF-hosted tools.
Is Apache Spark production ready?+
43,531 GitHub stars and continuous commit activity through mid-2026 indicate an actively maintained project, not something coasting on past adoption. Founding date and latest release version are not publicly available in this dataset, but Spark has shipped major versions (1.x through 3.x, with 4.
Who should use Apache Spark?+
Teams running batch ETL at a scale where single-node tools (Pandas, DuckDB) fall over — think multi-TB to PB daily volumes. Teams needing one engine across batch, streaming (Structured Streaming), and ML (MLlib) instead of stitching three separate systems together.
Who should not use Apache Spark?+
Small-data teams. If your datasets fit on one machine, Spark's overhead (JVM startup, cluster coordination, shuffle costs) will cost you more in ops time than it saves in compute. Use DuckDB or Polars instead. Low-latency streaming.
What are the alternatives to Apache Spark?+
Apache Flink — better fit for true low-latency, event-at-a-time streaming. Trino/Presto — better for interactive federated SQL queries, not built for heavy batch ETL or ML. DuckDB — better for single-node analytical workloads where a cluster is overkill.
How much does Apache Spark cost?+
Fully open source, Apache-2.0. No cost to run Spark itself. Real spend comes from infrastructure (compute clusters) and, optionally, managed platforms (Databricks, EMR, Dataproc, Synapse) that charge for orchestration, autoscaling, and support on top of the open-source engine.