#1
An open-source, event-driven orchestration and scheduling platform where data, AI and infrastructure workflows are defined declaratively in YAML and managed from a UI.
- GitHub stars
- 30k
- Last commit
- today
- Latest release
- v2.0.10
- Licence
- Apache-2.0
- Self-hosted
- Yes
go.kestra.io
Kestra is an orchestration platform, open source, for data pipelines, AI workflows and infrastructure automation. It brings scheduled and event-triggered automation together under one declarative interface that does not depend on a programming language, applying infrastructure-as-code practices to pipelines so that reliable workflows can be defined in a few lines of YAML.
Workflows can be built in the UI, written by a built-in AI Copilot, or generated from coding agents such as Claude Code and Cursor using agent skills, and everything can be kept as code with Git integration even when it was authored visually. A large plugin ecosystem connects it to external systems, and key concepts cover tasks, triggers and flows. The project is written in Java, with topics covering orchestration, high availability, pipeline-as-code and DevOps, and the 2.0 release adds new capabilities.
Kestra is Apache-2.0 licensed and can be self-hosted, with a quick start that gets a first workflow running in minutes. It targets data engineers, platform teams and automation developers who want an alternative to schedulers such as Airflow with a more declarative style.
Key features
- Declarative YAML workflow definitions
- Scheduled and event-driven triggers
- Visual editor and AI Copilot
- Git version control integration
- Large plugin ecosystem
- Agent skills for coding agents
Pricing: Free and open source under the Apache-2.0 license.
#2
An Apache workflow orchestration platform where data teams author, schedule and monitor pipelines as Python code, widely used for data engineering and ELT jobs.
- GitHub stars
- 47k
- Last commit
- today
- Latest release
- 3.4.0
- Licence
- Apache-2.0
- Self-hosted
- Yes
airflow.apache.org
Apache Airflow is a workflow platform where you write, schedule and watch over pipelines in code. Workflows are defined as Python code in the form of directed acyclic graphs, or DAGs, where each task and its dependencies are explicit, so pipelines can be versioned, tested and reviewed like any other software.
A scheduler runs tasks on a defined cadence and workers execute them, while a web interface shows the status of each run, lets you inspect logs and retry failures. Airflow is used heavily in data engineering, including data integration, ELT and ETL pipelines, data orchestration and machine learning workflows, and its topics cover data science and automation. It has a large ecosystem of providers and operators for connecting to databases, cloud services and other systems, and the 3.x line is the current series.
Airflow is an Apache Software Foundation project under the Apache-2.0 license and runs on your own infrastructure, from a single machine to Kubernetes. Several vendors offer managed Airflow services, but the project itself is self-hosted software. It suits data engineers who prefer code-first orchestration over drag-and-drop tools.
Key features
- Workflows defined as Python DAGs
- Scheduler for recurring and dependent tasks
- Web UI for monitoring runs and logs
- Retries and task dependency management
- Large provider ecosystem of integrations
- Runs on single machines or Kubernetes
Pricing: Free and open source under the Apache-2.0 license.
#3
An analytics engine for processing large datasets, with APIs in Scala, Java, Python and R and libraries for SQL, machine learning, graphs and streaming.
- GitHub stars
- 44k
- Last commit
- yesterday
- Licence
- Apache-2.0
- Self-hosted
- Yes
spark.apache.org
Apache Spark is an analytics engine that brings batch, SQL, machine learning, graph and streaming workloads together for processing data at scale. It is maintained by the Apache Software Foundation under the Apache-2.0 license. It is written mainly in Scala and offers high-level APIs in Scala, Java and Python, with an R API that is now deprecated, on top of an engine optimized to execute general computation graphs.
A set of higher-level tools is included: Spark SQL for queries and DataFrames, MLlib for machine learning, GraphX for graphs, Structured Streaming for streams, and a pandas-style API for existing pandas workloads. Together they let teams run batch analytics, machine learning and streaming jobs on a single engine and share code between them.
The README points to the project website for full documentation and programming guides, and it contains only basic setup and build instructions. Spark suits data engineers and data scientists who process datasets too large for a single machine, and organizations that want one framework for ETL, analytics and machine learning on clusters.
Key features
- High-level APIs in Scala, Java and Python
- Optimized engine for computation graphs
- Spark SQL and DataFrames
- MLlib for machine learning
- GraphX for graph analytics
- Structured Streaming for stream processing
- Pandas API on Spark
Pricing: Free and open source under the Apache-2.0 license.
#4
A Python workflow orchestration framework for turning scripts into resilient, observable data pipelines with scheduling, retries, caching and event-driven automation.
- GitHub stars
- 24k
- Last commit
- today
- Latest release
- 3.9.2
- Licence
- Apache-2.0
- Self-hosted
- Yes
- Hosted version
- Available
prefect.io
Prefect is a Python framework for orchestrating workflows and building data pipelines. It aims to be the simplest way to lift an ordinary script into a production workflow: you add flow and task decorators, and Prefect adds the machinery that makes pipelines dependable and visible.
That machinery includes scheduling, caching, retries and event-based automations, and flows can express dependencies and complex branching logic. The goal is dynamic pipelines that react to changes in the world and recover from unexpected failures. Run activity is tracked, and you can monitor it from a self-hosted Prefect server or from the managed Prefect Cloud dashboard. Prefect requires Python 3.10 or newer, and the documentation covers installation, quickstart, building and deploying workflows and agent setup.
Prefect is licensed under Apache-2.0. Topics list data engineering, ML ops and observability among its uses. It is aimed at data teams who prefer to write workflows as plain Python code, and who want a choice between running the orchestrator themselves or using a hosted control plane.
Key features
- Flows and tasks defined with Python decorators
- Scheduling and event-based automations
- Automatic retries and caching
- Complex branching and dependencies
- Self-hosted server or Prefect Cloud monitoring
- Deployment of workflows to infrastructure
Pricing: Prefect Cloud's Hobby tier is free. Starter is $100 per month and Team $100 per user per month, both billed monthly; Enterprise is custom and billed annually.
#5
An open-source data movement platform with hundreds of connectors for ELT pipelines from APIs, databases and files into warehouses, lakes and AI applications.
- GitHub stars
- 22k
- Last commit
- today
- Latest release
- v2.0.0
- Self-hosted
- Yes
- Hosted version
- Available
airbyte.com
Airbyte is an open-source data movement platform for moving data from APIs, databases and files into data warehouses, data lakes and AI applications. Its premise is that only an open-source project can reach the long tail of data sources while letting engineers tailor existing connectors, with the aim of moving data from any source to any destination.
It offers a catalog of 600 or more connectors for APIs, databases, warehouses, lakes and AI applications, with warehouse destinations such as BigQuery, Redshift and Snowflake, and topics also mention change data capture. The README distinguishes products by job: Airbyte Open Source or Airbyte Cloud for ELT and ETL into warehouses, lakes or databases, and Airbyte Agents, a managed data and context layer, plus an open-source Agent SDK for giving AI agents and MCP clients real-time access to business data.
Airbyte is written in Python and Java, and its license is listed as 'Other' on GitHub because the project uses a mix of licenses, so review the terms for your use. It can be self-hosted or run as the managed Airbyte Cloud, and suits data teams that need many integrations without writing each pipeline by hand.
Key features
- 600+ source and destination connectors
- ELT pipelines into warehouses and lakes
- Change data capture support
- Customizable open-source connectors
- Agent SDK for AI agent data access
- Self-hosted or Airbyte Cloud
Pricing: The open-source Core edition is free to self-host. Managed cloud starts at $20 a month (Standard) or $189 a month for 40 credits (Plus); Pro and Enterprise Flex are custom. 30-day trial.
#6
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams, designed to run in all common cluster environments.
- GitHub stars
- 26k
- Last commit
- yesterday
- Licence
- Apache-2.0
- Self-hosted
- Yes
flink.apache.org
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. It has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Flink provides correctness guarantees including exactly-once state consistency, event-time processing, and sophisticated late data handling. It offers layered APIs such as SQL on stream and batch data, DataStream API, and ProcessFunction for time and state.
Key features
- Exactly-once state consistency
- Event-time processing
- Sophisticated late data handling
- Layered APIs including SQL, DataStream API, and ProcessFunction
- Flexible deployment and high-availability setup
- Scale-out architecture with support for very large state and incremental checkpoints
#7
A Python data orchestration platform for building, scheduling and observing data pipelines around the data assets they produce; Dagster is now part of Prefect.
- GitHub stars
- 16k
- Last commit
- today
- Latest release
- 1.13.30
- Licence
- Apache-2.0
- Self-hosted
- Yes
- Hosted version
- Available
dagster.io
Dagster is an orchestration platform for developing, running and observing data assets. Instead of thinking only in terms of tasks, teams define the tables, files and models their pipelines produce, and Dagster tracks how they are built, scheduled and monitored. It is written in Python and aimed at data engineering, analytics and machine learning workloads.
The product surface covers data orchestration, a data catalog, data quality checks, cost insights and many integrations, with an enterprise offering for larger organizations. Comparison pages on the website position it against Airflow, dbt Cloud, Azure Data Factory and AWS Step Functions, and topics include ETL, scheduling and MLOps. The homepage now announces that Dagster is part of Prefect, and points visitors looking for agentic orchestration or MCP support to Prefect.
The open-source core is Apache-2.0 licensed and can be self-hosted, while a managed cloud and enterprise plans are offered with pricing on the company site. Dagster University and documentation help with learning. It suits data teams that want a software-engineering-style approach to pipelines.
Key features
- Asset-centric data orchestration
- Scheduling and monitoring of pipelines
- Data catalog and data quality features
- Cost insights for pipelines
- Integrations with common data tools
- Pipelines developed in Python
Pricing: Dagster+ Solo is $10 and Starter $100 per month, each plus per-credit usage charges, with a 30-day free trial. Pro is quoted through sales.
#8
A data transformation framework that lets analysts and engineers build analytics models with software-engineering practices; v2.0 is a ground-up rewrite in Rust.
- GitHub stars
- 14k
- Last commit
- today
- Latest release
- v1.12.7
- Licence
- Apache-2.0
- Self-hosted
- Yes
- Hosted version
- Available
getdbt.com
dbt is a framework for transforming data in a warehouse using the same practices that software engineers apply to applications. Analysts and analytics engineers write models, and dbt parses, compiles and runs the project so that transformations are version-controlled and repeatable.
The repository now holds the Apache 2.0 source of dbt v2.0, a ground-up rewrite in Rust. According to its README, v2.0 parses and compiles projects much faster than v1, enforces a stricter language specification at parse time, produces Parquet artifacts alongside the existing JSON ones, and ships as one standalone executable, so no Python installation is needed. The local documentation experience was rebuilt around those artifacts to handle large projects.
The Apache 2.0 code is available to everyone, while the packaged dbt distribution adds dbt-specific customizations under a separate dbt product license. v2.0 supports macOS and Linux on x86-64 and ARM. The v1 Python implementation continues on the 1.latest branch. dbt Labs also offers dbt as a commercial cloud platform.
Key features
- Data transformation using software engineering practices
- Faster parsing and compiling in v2.0
- Strict language specification checks at parse time
- Parquet artifacts for querying project metadata
- Single binary without a Python runtime
- Rebuilt local documentation experience
Pricing: A single-seat Developer plan to start free. Starter costs $100 per user per month; Enterprise and Enterprise+ are custom. A trial is available and add-ons are billed on usage.
#9
Apache Beam is an open-source programming model for batch and streaming data processing whose pipelines run on multiple execution engines.
- GitHub stars
- 8.7k
- Last commit
- yesterday
- Latest release
- v2.77.0
- Licence
- Apache-2.0
beam.apache.org
Apache Beam provides a single programming model for both batch and streaming data processing. Pipelines read from a range of supported sources, apply business logic, and write results to common data sinks. Quickstarts are offered for Java, Python and Go.
Beam pipelines can execute on multiple execution environments, called runners, which the project presents as a way to avoid lock-in. It is extensible, with projects such as TensorFlow Extended and Apache Hop built on top of it, and it supports multi-language pipelines. The Beam Playground lets people try transforms and examples without installing anything locally.
The project publishes case studies from several organisations. LinkedIn is described as running streaming infrastructure on Beam, OCTO used it to move a French grocery retailer to streaming processing of transactional data, HSBC used it as a computational platform and risk engine, and Booking.com uses it for global ad bidding. The site also invites other users to share their own story.
Key features
- One programming model for batch and streaming
- Pipelines run on several runners
- Reads from many sources on-prem or in the cloud
- Extended by projects such as TensorFlow Extended
- Interactive Beam Playground
#10
An open-source ETL and ELT platform built on DuckDB that you deploy yourself, with visual or SQL pipelines, dbt, CDC, data quality and an MCP server.
- GitHub stars
- 1.6k
- Last commit
- today
- Latest release
- v0.8.0
- Licence
- Apache-2.0
- Self-hosted
- Yes
duckle.org
Duckle is an open-source ETL and ELT platform for teams that want their data pipelines running on their own infrastructure. You build pipelines on a visual canvas, in Python or in SQL, then ship the same file to your own server or cloud account, where a headless runner executes it on a schedule.
Pipelines compile to SQL on DuckDB and use all the cores available, so a larger machine runs them faster. The platform lists support for a large catalog of components, dbt, change data capture, data quality checks, reverse ETL and lineage, along with a web console, roles, an audit trail and an MCP server so AI agents such as Claude or Cursor can work with it. Each pipeline is a single file that can live in git.
Duckle is written in Rust and licensed under Apache-2.0, and it runs headless in Docker or on a plain server, with Kubernetes among its topics. It states there is no vendor cloud and no per-row billing. It is an independent project by SlothFlowLabs and is not affiliated with DuckDB Labs or MotherDuck. It suits data engineers who want a self-hosted alternative to managed pipeline services.
Key features
- Visual canvas, Python, or SQL pipeline authoring
- Runs on DuckDB across all CPU cores
- dbt, CDC, and reverse ETL support
- Data quality checks and lineage
- Web console with roles and audit trail
- MCP server for AI agents
Pricing: Free and open source under the Apache-2.0 licence, with no vendor cloud or per-row billing according to the README.
2 more AWS Glue alternatives
AWS Glue alternatives: questions
- What is the best open-source alternative to AWS Glue?
- Kestra is the top-ranked open-source alternative to AWS Glue on Enlisted: An open-source, event-driven orchestration and scheduling platform where data, AI and infrastructure workflows are defined declaratively in YAML and managed from a UI. Other strong options are Airflow, Apache Spark, Prefect and Airbyte.
- What are the best commercial alternatives to AWS Glue?
- Talend, Azure Data Factory, Matillion, Google Cloud Dataflow and Astronomer are among the hosted, closed-source alternatives to AWS Glue listed on Enlisted, which lists 20 in total. Compare their pricing and features on each tool page.
- Are these AWS Glue alternatives free?
- All 12 are open source, so the code is free to use under its licence, and all of them can be self-hosted on your own server or computer. 5 also offer a paid or managed cloud version if you'd rather not host it yourself.
- How is this list of AWS Glue alternatives ranked?
- By a score built from GitHub stars, star growth over the last 30 days and how recently the code changed. 12 of these projects shipped code in the last 30 days. Data is refreshed daily, and nobody can pay to move up.