Open-source Databricks alternatives

Apache Spark is an analytics engine that brings batch, SQL, machine learning, graph and streaming workloads together for processing data at scale. It is maintained by the Apache Software Foundation under the Apache-2.0 license. It is written mainly in Scala and offers high-level APIs in Scala, Java and Python, with an R API that is now deprecated, on top of an engine optimized to execute general computation graphs.

A set of higher-level tools is included: Spark SQL for queries and DataFrames, MLlib for machine learning, GraphX for graphs, Structured Streaming for streams, and a pandas-style API for existing pandas workloads. Together they let teams run batch analytics, machine learning and streaming jobs on a single engine and share code between them.

The README points to the project website for full documentation and programming guides, and it contains only basic setup and build instructions. Spark suits data engineers and data scientists who process datasets too large for a single machine, and organizations that want one framework for ETL, analytics and machine learning on clusters.

Apache Zeppelin is a web-based notebook for interactive data analytics. You build data-driven, collaborative documents using SQL, Scala and other languages, and the notebook-style editor supports real-time collaboration between users.

Core features include support for many languages through a pluggable interpreter architecture with process isolation, including Spark, Flink, Python, SQL and Shell and more than 20 interpreters, built-in visualization and dynamic forms, and notebook scheduling with cron. Flexible deployment options are local, Docker, Kubernetes and YARN. Topics place it in the big data ecosystem alongside Spark, Flink, NoSQL and databases.

Zeppelin is an Apache Software Foundation project written in Java and JavaScript and licensed under Apache-2.0. You install it from a binary package or build from source, with user guides, mailing lists and a Jira issue tracker. It suits data engineers and analysts who work on cluster-based data platforms and want shareable, scheduled notebooks.

Apache Cloudberry, currently in the Apache incubator, is an open-source massively parallel processing database. It was created by original developers of Greenplum Database and evolves from the open-source version of Pivotal Greenplum, but uses a newer PostgreSQL kernel and adds enterprise capabilities. It is described as an open-source alternative to Greenplum Database.

It can serve as a data warehouse and is also intended for large-scale analytics and AI or machine learning workloads, distributing queries across many nodes. Because it is based on PostgreSQL, it uses familiar SQL. The main repository sits alongside ecosystem repositories for the website, a backup utility, Go libraries and the Platform Extension Framework (PXF) for connecting to external data.

Cloudberry is written in C and licensed under Apache-2.0. You can build it from source on Linux, including RHEL, Rocky Linux and Ubuntu, and macOS, or try it quickly through a Docker-based sandbox. Community help is available through Slack and GitHub Discussions. It suits data engineering teams running large analytical workloads that want an open-source MPP engine they can operate themselves.

Pachyderm is a data pipeline platform built for data engineering teams that need to automate complex transformations across large and varied datasets. It positions itself as a CI/CD engine for data, combining pipeline automation with built-in version control and lineage so that every change to data and every pipeline run can be traced.

Pipelines trigger automatically when Pachyderm detects a change in input data, and each stage can be written in any language since Pachyderm treats containers as the unit of processing. It tracks immutable data lineage and versioning for any data type, scales and parallelizes processing automatically on top of Kubernetes, and stores data in standard object stores with automatic deduplication. It runs on all major cloud providers as well as on-premises.

Pachyderm is written in Go and released under the Apache-2.0 license. It can be run locally for evaluation or deployed on AWS, GCE or Azure in a short setup, and the project provides tutorials, example projects and case studies through its documentation.

Dremio is a data analytics platform aimed at organizations that want to query and analyze large volumes of data without first moving or copying it into a separate, dedicated analytics database. It positions itself as filling a gap between raw data storage and the tools people need to get value out of that data.

The open-source edition can be built and run locally, exposing a web UI at localhost:9047 once started, or installed as a production tarball for server deployment, with an embedded mode also available. Building it requires a specific combination of JDK versions (21 as default, with 17 and 11 configured in the Maven toolchain for certain tests) plus Maven, reflecting a substantial, enterprise-grade Java codebase.

The README notes that to provide the best possible experience, the build includes some dependencies distributed under non-open-source licenses, so the fully open-source experience may differ slightly depending on build configuration. Dremio is written in Java and the open-source edition is released under the Apache-2.0 license, with full documentation hosted separately.

SnappyData is a distributed, memory-optimized analytics database that combines an in-memory hybrid database with Apache Spark. In one cluster it provides analytic query processing, mutable data with transactions, access to many big data sources and stream processing. TIBCO also marketed it as ComputeDB.

A typical use is interactive analytics over large datasets with little pre-processing, avoiding the need to pre-aggregate data or build cubes for ad hoc visual analytics. It manages data in memory, generates code with vectorization optimizations and makes use of multi-core CPUs to keep response times low. The README stresses it is not a data warehouse replacement but a compute and caching cluster that augments warehouses and data lakes.

The repository is provided for legacy users and informational purposes only: TIBCO states it provides no updates, including security updates, and that the code may contain vulnerabilities. The TIBCO code is under the Apache License 2.0, although the repository metadata lists the license as Other. Evaluate it carefully before any new deployment.