Open-source Google Cloud Dataflow alternatives

Apache Spark is an analytics engine that brings batch, SQL, machine learning, graph and streaming workloads together for processing data at scale. It is maintained by the Apache Software Foundation under the Apache-2.0 license. It is written mainly in Scala and offers high-level APIs in Scala, Java and Python, with an R API that is now deprecated, on top of an engine optimized to execute general computation graphs.

A set of higher-level tools is included: Spark SQL for queries and DataFrames, MLlib for machine learning, GraphX for graphs, Structured Streaming for streams, and a pandas-style API for existing pandas workloads. Together they let teams run batch analytics, machine learning and streaming jobs on a single engine and share code between them.

The README points to the project website for full documentation and programming guides, and it contains only basic setup and build instructions. Spark suits data engineers and data scientists who process datasets too large for a single machine, and organizations that want one framework for ETL, analytics and machine learning on clusters.

RisingWave is an event streaming platform, positioned for agentic AI and real-time applications, that continuously takes in data, transforms it and serves it. It is meant to replace the familiar chain of Debezium (change data capture), Kafka (transport), Flink (processing) and a serving database with one system, removing the latency and operational work of each hop.

Sources include webhooks from SaaS applications, native change data capture from PostgreSQL and MySQL, event streams such as Kafka, Pulsar and Kinesis, and batch data from S3 and warehouses, all unified under a SQL interface where streams and tables can be joined. Processing is incremental: if upstream data changes, just the affected results get recomputed, which keeps materialized views up to date without full recomputation. Results are queryable directly, and the topics mention PostgreSQL compatibility and Apache Iceberg.

RisingWave is written in Rust and licensed under Apache-2.0. It can be tried with Docker or Kubernetes using the quick start guide, and it suits data engineers who want streaming pipelines defined in SQL instead of several separate services.