Open-source Azure Data Factory alternatives

Apache Airflow is a workflow platform where you write, schedule and watch over pipelines in code. Workflows are defined as Python code in the form of directed acyclic graphs, or DAGs, where each task and its dependencies are explicit, so pipelines can be versioned, tested and reviewed like any other software.

A scheduler runs tasks on a defined cadence and workers execute them, while a web interface shows the status of each run, lets you inspect logs and retry failures. Airflow is used heavily in data engineering, including data integration, ELT and ETL pipelines, data orchestration and machine learning workflows, and its topics cover data science and automation. It has a large ecosystem of providers and operators for connecting to databases, cloud services and other systems, and the 3.x line is the current series.

Airflow is an Apache Software Foundation project under the Apache-2.0 license and runs on your own infrastructure, from a single machine to Kubernetes. Several vendors offer managed Airflow services, but the project itself is self-hosted software. It suits data engineers who prefer code-first orchestration over drag-and-drop tools.

Kestra is an orchestration platform, open source, for data pipelines, AI workflows and infrastructure automation. It brings scheduled and event-triggered automation together under one declarative interface that does not depend on a programming language, applying infrastructure-as-code practices to pipelines so that reliable workflows can be defined in a few lines of YAML.

Workflows can be built in the UI, written by a built-in AI Copilot, or generated from coding agents such as Claude Code and Cursor using agent skills, and everything can be kept as code with Git integration even when it was authored visually. A large plugin ecosystem connects it to external systems, and key concepts cover tasks, triggers and flows. The project is written in Java, with topics covering orchestration, high availability, pipeline-as-code and DevOps, and the 2.0 release adds new capabilities.

Kestra is Apache-2.0 licensed and can be self-hosted, with a quick start that gets a first workflow running in minutes. It targets data engineers, platform teams and automation developers who want an alternative to schedulers such as Airflow with a more declarative style.

Prefect is a Python framework for orchestrating workflows and building data pipelines. It aims to be the simplest way to lift an ordinary script into a production workflow: you add flow and task decorators, and Prefect adds the machinery that makes pipelines dependable and visible.

That machinery includes scheduling, caching, retries and event-based automations, and flows can express dependencies and complex branching logic. The goal is dynamic pipelines that react to changes in the world and recover from unexpected failures. Run activity is tracked, and you can monitor it from a self-hosted Prefect server or from the managed Prefect Cloud dashboard. Prefect requires Python 3.10 or newer, and the documentation covers installation, quickstart, building and deploying workflows and agent setup.

Prefect is licensed under Apache-2.0. Topics list data engineering, ML ops and observability among its uses. It is aimed at data teams who prefer to write workflows as plain Python code, and who want a choice between running the orchestrator themselves or using a hosted control plane.

Airbyte is an open-source data movement platform for moving data from APIs, databases and files into data warehouses, data lakes and AI applications. Its premise is that only an open-source project can reach the long tail of data sources while letting engineers tailor existing connectors, with the aim of moving data from any source to any destination.

It offers a catalog of 600 or more connectors for APIs, databases, warehouses, lakes and AI applications, with warehouse destinations such as BigQuery, Redshift and Snowflake, and topics also mention change data capture. The README distinguishes products by job: Airbyte Open Source or Airbyte Cloud for ELT and ETL into warehouses, lakes or databases, and Airbyte Agents, a managed data and context layer, plus an open-source Agent SDK for giving AI agents and MCP clients real-time access to business data.

Airbyte is written in Python and Java, and its license is listed as 'Other' on GitHub because the project uses a mix of licenses, so review the terms for your use. It can be self-hosted or run as the managed Airbyte Cloud, and suits data teams that need many integrations without writing each pipeline by hand.

Dagster is an orchestration platform for developing, running and observing data assets. Instead of thinking only in terms of tasks, teams define the tables, files and models their pipelines produce, and Dagster tracks how they are built, scheduled and monitored. It is written in Python and aimed at data engineering, analytics and machine learning workloads.

The product surface covers data orchestration, a data catalog, data quality checks, cost insights and many integrations, with an enterprise offering for larger organizations. Comparison pages on the website position it against Airflow, dbt Cloud, Azure Data Factory and AWS Step Functions, and topics include ETL, scheduling and MLOps. The homepage now announces that Dagster is part of Prefect, and points visitors looking for agentic orchestration or MCP support to Prefect.

The open-source core is Apache-2.0 licensed and can be self-hosted, while a managed cloud and enterprise plans are offered with pricing on the company site. Dagster University and documentation help with learning. It suits data teams that want a software-engineering-style approach to pipelines.

Mage, called Mage OSS in its README, is a self-hosted development environment for building data pipelines. It is meant for teams that automate ETL tasks, design data flows or orchestrate transformations, and it presents the work in a notebook-style interface made of modular blocks of code.

Pipelines are written block by block in Python, SQL or R. Jobs can be run manually or on a schedule, including cron expressions, and prebuilt connectors reach databases, APIs and cloud storage. Debugging is visual, with logs, live data previews and execution you can follow step by step, and dbt models can be built and run inside Mage. Topics on the repository also reference Spark, reverse ETL and machine learning workloads.

Mage OSS is licensed under Apache-2.0 and installs with Docker, pip or conda, with no cloud account required. For larger deployments the vendor offers Mage Pro, a commercial platform that adds enterprise orchestration, collaboration and AI-assisted workflows.

Bruin is a data pipeline tool that brings data ingestion, transformation and data quality into a single framework. Instead of stitching together separate tools for loading, modeling and testing, you describe a pipeline with SQL, Python or R assets and run it against the major data platforms, such as BigQuery and Snowflake.

Pipelines can ingest data from different sources, materialize tables and views, and build incremental tables. Python assets run in isolated environments using uv, Jinja templating cuts down on repetition, and built-in quality checks validate results. A dry-run mode validates a pipeline end to end, secrets are injected through environment variables, and a VS Code extension supports day-to-day development.

Bruin is written in Go, licensed under Apache-2.0 and distributed as a command-line tool that is easy to install. It runs on your local machine, an EC2 instance or GitHub Actions, so no dedicated orchestration server is required. It suits data engineers and analytics teams who want pipelines defined as code in a repository.

Duckle is an open-source ETL and ELT platform for teams that want their data pipelines running on their own infrastructure. You build pipelines on a visual canvas, in Python or in SQL, then ship the same file to your own server or cloud account, where a headless runner executes it on a schedule.

Pipelines compile to SQL on DuckDB and use all the cores available, so a larger machine runs them faster. The platform lists support for a large catalog of components, dbt, change data capture, data quality checks, reverse ETL and lineage, along with a web console, roles, an audit trail and an MCP server so AI agents such as Claude or Cursor can work with it. Each pipeline is a single file that can live in git.

Duckle is written in Rust and licensed under Apache-2.0, and it runs headless in Docker or on a plain server, with Kubernetes among its topics. It states there is no vendor cloud and no per-row billing. It is an independent project by SlothFlowLabs and is not affiliated with DuckDB Labs or MotherDuck. It suits data engineers who want a self-hosted alternative to managed pipeline services.