Open-source AWS Glue alternatives

Apache Spark is an analytics engine that brings batch, SQL, machine learning, graph and streaming workloads together for processing data at scale. It is maintained by the Apache Software Foundation under the Apache-2.0 license. It is written mainly in Scala and offers high-level APIs in Scala, Java and Python, with an R API that is now deprecated, on top of an engine optimized to execute general computation graphs.

A set of higher-level tools is included: Spark SQL for queries and DataFrames, MLlib for machine learning, GraphX for graphs, Structured Streaming for streams, and a pandas-style API for existing pandas workloads. Together they let teams run batch analytics, machine learning and streaming jobs on a single engine and share code between them.

The README points to the project website for full documentation and programming guides, and it contains only basic setup and build instructions. Spark suits data engineers and data scientists who process datasets too large for a single machine, and organizations that want one framework for ETL, analytics and machine learning on clusters.

Airbyte is an open-source data movement platform for moving data from APIs, databases and files into data warehouses, data lakes and AI applications. Its premise is that only an open-source project can reach the long tail of data sources while letting engineers tailor existing connectors, with the aim of moving data from any source to any destination.

It offers a catalog of 600 or more connectors for APIs, databases, warehouses, lakes and AI applications, with warehouse destinations such as BigQuery, Redshift and Snowflake, and topics also mention change data capture. The README distinguishes products by job: Airbyte Open Source or Airbyte Cloud for ELT and ETL into warehouses, lakes or databases, and Airbyte Agents, a managed data and context layer, plus an open-source Agent SDK for giving AI agents and MCP clients real-time access to business data.

Airbyte is written in Python and Java, and its license is listed as 'Other' on GitHub because the project uses a mix of licenses, so review the terms for your use. It can be self-hosted or run as the managed Airbyte Cloud, and suits data teams that need many integrations without writing each pipeline by hand.

Mage, called Mage OSS in its README, is a self-hosted development environment for building data pipelines. It is meant for teams that automate ETL tasks, design data flows or orchestrate transformations, and it presents the work in a notebook-style interface made of modular blocks of code.

Pipelines are written block by block in Python, SQL or R. Jobs can be run manually or on a schedule, including cron expressions, and prebuilt connectors reach databases, APIs and cloud storage. Debugging is visual, with logs, live data previews and execution you can follow step by step, and dbt models can be built and run inside Mage. Topics on the repository also reference Spark, reverse ETL and machine learning workloads.

Mage OSS is licensed under Apache-2.0 and installs with Docker, pip or conda, with no cloud account required. For larger deployments the vendor offers Mage Pro, a commercial platform that adds enterprise orchestration, collaboration and AI-assisted workflows.

Bruin is a data pipeline tool that brings data ingestion, transformation and data quality into a single framework. Instead of stitching together separate tools for loading, modeling and testing, you describe a pipeline with SQL, Python or R assets and run it against the major data platforms, such as BigQuery and Snowflake.

Pipelines can ingest data from different sources, materialize tables and views, and build incremental tables. Python assets run in isolated environments using uv, Jinja templating cuts down on repetition, and built-in quality checks validate results. A dry-run mode validates a pipeline end to end, secrets are injected through environment variables, and a VS Code extension supports day-to-day development.

Bruin is written in Go, licensed under Apache-2.0 and distributed as a command-line tool that is easy to install. It runs on your local machine, an EC2 instance or GitHub Actions, so no dedicated orchestration server is required. It suits data engineers and analytics teams who want pipelines defined as code in a repository.

Duckle is an open-source ETL and ELT platform for teams that want their data pipelines running on their own infrastructure. You build pipelines on a visual canvas, in Python or in SQL, then ship the same file to your own server or cloud account, where a headless runner executes it on a schedule.

Pipelines compile to SQL on DuckDB and use all the cores available, so a larger machine runs them faster. The platform lists support for a large catalog of components, dbt, change data capture, data quality checks, reverse ETL and lineage, along with a web console, roles, an audit trail and an MCP server so AI agents such as Claude or Cursor can work with it. Each pipeline is a single file that can live in git.

Duckle is written in Rust and licensed under Apache-2.0, and it runs headless in Docker or on a plain server, with Kubernetes among its topics. It states there is no vendor cloud and no per-row billing. It is an independent project by SlothFlowLabs and is not affiliated with DuckDB Labs or MotherDuck. It suits data engineers who want a self-hosted alternative to managed pipeline services.