7 open-source alternatives ranked by real activity

Open-source Google Cloud Dataflow alternatives

A curated, ranked list of the 7 best open-source alternatives to Google Cloud Dataflow.

The best open-source alternative to Google Cloud Dataflow is Airflow. If that doesn't suit you, other good options are Apache Spark, Kafka, Flink and RisingWave.

Commercial alternatives to Google Cloud Dataflow include AWS Glue, Amazon EMR, Amazon Kinesis and Azure Data Factory. See all 12.

Google Cloud Dataflow alternatives are mainly data pipeline & ETL tools, but some are also media & streaming tools. 7 of them shipped code in the last 30 days, 7 can be self-hosted, and 7 use a permissive licence.

Last updated November 13, 2026 · ranked by GitHub stars, growth and recent commits

Airflow

An Apache workflow orchestration platform where data teams author, schedule and monitor pipelines as Python code, widely used for data engineering and ELT jobs.

GitHub stars
47k
Last commit
today
Latest release
3.4.0
Licence
Apache-2.0
Self-hosted
Yes
airflow.apache.orgAirflow homepage screenshot

Apache Airflow is a workflow platform where you write, schedule and watch over pipelines in code. Workflows are defined as Python code in the form of directed acyclic graphs, or DAGs, where each task and its dependencies are explicit, so pipelines can be versioned, tested and reviewed like any other software.

A scheduler runs tasks on a defined cadence and workers execute them, while a web interface shows the status of each run, lets you inspect logs and retry failures. Airflow is used heavily in data engineering, including data integration, ELT and ETL pipelines, data orchestration and machine learning workflows, and its topics cover data science and automation. It has a large ecosystem of providers and operators for connecting to databases, cloud services and other systems, and the 3.x line is the current series.

Airflow is an Apache Software Foundation project under the Apache-2.0 license and runs on your own infrastructure, from a single machine to Kubernetes. Several vendors offer managed Airflow services, but the project itself is self-hosted software. It suits data engineers who prefer code-first orchestration over drag-and-drop tools.

Key features

  • Workflows defined as Python DAGs
  • Scheduler for recurring and dependent tasks
  • Web UI for monitoring runs and logs
  • Retries and task dependency management
  • Large provider ecosystem of integrations
  • Runs on single machines or Kubernetes

Pricing: Free and open source under the Apache-2.0 license.

Read more about AirflowWebsiteGitHub#1 Google Cloud Dataflow alternative

Apache Spark

An analytics engine for processing large datasets, with APIs in Scala, Java, Python and R and libraries for SQL, machine learning, graphs and streaming.

GitHub stars
44k
Last commit
yesterday
Licence
Apache-2.0
Self-hosted
Yes
spark.apache.orgApache Spark homepage screenshot

Apache Spark is an analytics engine that brings batch, SQL, machine learning, graph and streaming workloads together for processing data at scale. It is maintained by the Apache Software Foundation under the Apache-2.0 license. It is written mainly in Scala and offers high-level APIs in Scala, Java and Python, with an R API that is now deprecated, on top of an engine optimized to execute general computation graphs.

A set of higher-level tools is included: Spark SQL for queries and DataFrames, MLlib for machine learning, GraphX for graphs, Structured Streaming for streams, and a pandas-style API for existing pandas workloads. Together they let teams run batch analytics, machine learning and streaming jobs on a single engine and share code between them.

The README points to the project website for full documentation and programming guides, and it contains only basic setup and build instructions. Spark suits data engineers and data scientists who process datasets too large for a single machine, and organizations that want one framework for ETL, analytics and machine learning on clusters.

Key features

  • High-level APIs in Scala, Java and Python
  • Optimized engine for computation graphs
  • Spark SQL and DataFrames
  • MLlib for machine learning
  • GraphX for graph analytics
  • Structured Streaming for stream processing
  • Pandas API on Spark

Pricing: Free and open source under the Apache-2.0 license.

Read more about Apache SparkWebsiteGitHub#2 Google Cloud Dataflow alternative

Kafka

Apache Kafka is an open-source platform for distributed event streaming, used in data pipelines, streaming analytics and data integration.

GitHub stars
34k
Last commit
yesterday
Licence
Apache-2.0
Self-hosted
Yes
kafka.apache.orgKafka homepage screenshot

Apache Kafka is a platform for distributed event streaming. Teams use it to move and process streams of events for data pipelines, streaming analytics, data integration and business-critical applications. The code lives in the Apache Software Foundation repository, is written mainly in Java with Scala for parts of the broker, and is released under the Apache-2.0 license.

Kafka is installed software that you run yourself. The project builds and tests against recent Java versions, supports Scala 2.13 only, and documents how to build a JAR and start a broker through its quickstart. The repository also contains client and streams modules, along with test tooling for unit and integration runs. Because Kafka is operated as infrastructure, adopters typically run clusters on their own servers or on Kubernetes, though managed services from other vendors exist.

Key features

  • Distributed event streaming platform
  • Durable publish and subscribe topics
  • Client libraries and a streams module
  • Suited to data pipelines and integration
  • Runs on Java with Scala 2.13
  • Quickstart for building and running a broker

Pricing: Free and open source under the Apache-2.0 license.

Read more about KafkaWebsiteGitHub#3 Google Cloud Dataflow alternative

RisingWave

An open-source event streaming platform that ingests data from databases, streams and webhooks, processes it incrementally with SQL and serves fresh results at low latency.

GitHub stars
9.4k
Last commit
today
Latest release
v3.1.1
Licence
Apache-2.0
Self-hosted
Yes
go.risingwave.comRisingWave homepage screenshot

RisingWave is an event streaming platform, positioned for agentic AI and real-time applications, that continuously takes in data, transforms it and serves it. It is meant to replace the familiar chain of Debezium (change data capture), Kafka (transport), Flink (processing) and a serving database with one system, removing the latency and operational work of each hop.

Sources include webhooks from SaaS applications, native change data capture from PostgreSQL and MySQL, event streams such as Kafka, Pulsar and Kinesis, and batch data from S3 and warehouses, all unified under a SQL interface where streams and tables can be joined. Processing is incremental: if upstream data changes, just the affected results get recomputed, which keeps materialized views up to date without full recomputation. Results are queryable directly, and the topics mention PostgreSQL compatibility and Apache Iceberg.

RisingWave is written in Rust and licensed under Apache-2.0. It can be tried with Docker or Kubernetes using the quick start guide, and it suits data engineers who want streaming pipelines defined in SQL instead of several separate services.

Key features

  • Ingestion from CDC, Kafka, webhooks and batch sources
  • Incremental stream processing in SQL
  • Always up-to-date materialized views
  • Low-latency serving of query results
  • Interface compatible with PostgreSQL clients
  • Docker and Kubernetes deployment

Pricing: Free and open source under the Apache-2.0 licence.

Read more about RisingWaveWebsiteGitHub#5 Google Cloud Dataflow alternative

Beam

Apache Beam is an open-source programming model for batch and streaming data processing whose pipelines run on multiple execution engines.

GitHub stars
8.7k
Last commit
yesterday
Latest release
v2.77.0
Licence
Apache-2.0
beam.apache.orgBeam homepage screenshot

Apache Beam provides a single programming model for both batch and streaming data processing. Pipelines read from a range of supported sources, apply business logic, and write results to common data sinks. Quickstarts are offered for Java, Python and Go.

Beam pipelines can execute on multiple execution environments, called runners, which the project presents as a way to avoid lock-in. It is extensible, with projects such as TensorFlow Extended and Apache Hop built on top of it, and it supports multi-language pipelines. The Beam Playground lets people try transforms and examples without installing anything locally.

The project publishes case studies from several organisations. LinkedIn is described as running streaming infrastructure on Beam, OCTO used it to move a French grocery retailer to streaming processing of transactional data, HSBC used it as a computational platform and risk engine, and Booking.com uses it for global ad bidding. The site also invites other users to share their own story.

Key features

  • One programming model for batch and streaming
  • Pipelines run on several runners
  • Reads from many sources on-prem or in the cloud
  • Extended by projects such as TensorFlow Extended
  • Interactive Beam Playground
Read more about BeamWebsiteGitHub#6 Google Cloud Dataflow alternative

Gobblin

Apache Gobblin is a distributed data integration framework for ingestion, replication and lifecycle management of streaming and batch data.

GitHub stars
2.3k
Last commit
29 days ago
Latest release
gobblin_0.11.0
Licence
Apache-2.0
gobblin.apache.orgGobblin homepage screenshot

Apache Gobblin is a distributed framework for big data integration. It simplifies common work such as data ingestion, replication, organisation and lifecycle management, for both streaming and batch data ecosystems. It became an Apache Top Level Project in January 2021.

It can run in several execution modes. Standalone mode runs as a single application on one machine and also supports an embedded mode. MapReduce mode runs as a MapReduce application on multiple Hadoop versions, with Azkaban supported for launching jobs. A cluster mode using YARN has primary and worker nodes and supports high availability.

A cloud mode runs as an elastic cluster on a public cloud, also with high availability. The news list on the site shows releases from 0.13.0-incubating in 2018 up to 0.17.0 in August 2023, and the project is published by the Apache Software Foundation under the Apache licence.

Key features

  • Data ingestion, replication and lifecycle management
  • Handles both streaming and batch data
  • Standalone mode on a single machine
  • MapReduce mode across Hadoop versions
  • Elastic cluster mode on public cloud
Read more about GobblinWebsiteGitHub#7 Google Cloud Dataflow alternative

Commercial Google Cloud Dataflow alternatives

Hosted, closed-source products that compete with Google Cloud Dataflow.

Google Cloud Dataflow alternatives: questions

What is the best open-source alternative to Google Cloud Dataflow?
Airflow is the top-ranked open-source alternative to Google Cloud Dataflow on Enlisted: An Apache workflow orchestration platform where data teams author, schedule and monitor pipelines as Python code, widely used for data engineering and ELT jobs. Other strong options are Apache Spark, Kafka, Flink and RisingWave.
What are the best commercial alternatives to Google Cloud Dataflow?
AWS Glue, Amazon EMR, Amazon Kinesis, Azure Data Factory and Astronomer are among the hosted, closed-source alternatives to Google Cloud Dataflow listed on Enlisted, which lists 12 in total. Compare their pricing and features on each tool page.
Are these Google Cloud Dataflow alternatives free?
All 7 are open source, so the code is free to use under its licence, and all of them can be self-hosted on your own server or computer.
How is this list of Google Cloud Dataflow alternatives ranked?
By a score built from GitHub stars, star growth over the last 30 days and how recently the code changed. 7 of these projects shipped code in the last 30 days. Data is refreshed daily, and nobody can pay to move up.

People also look for alternatives to…

View all