Open-source Amazon EMR alternatives

Apache Spark is an analytics engine that brings batch, SQL, machine learning, graph and streaming workloads together for processing data at scale. It is maintained by the Apache Software Foundation under the Apache-2.0 license. It is written mainly in Scala and offers high-level APIs in Scala, Java and Python, with an R API that is now deprecated, on top of an engine optimized to execute general computation graphs.

A set of higher-level tools is included: Spark SQL for queries and DataFrames, MLlib for machine learning, GraphX for graphs, Structured Streaming for streams, and a pandas-style API for existing pandas workloads. Together they let teams run batch analytics, machine learning and streaming jobs on a single engine and share code between them.

The README points to the project website for full documentation and programming guides, and it contains only basic setup and build instructions. Spark suits data engineers and data scientists who process datasets too large for a single machine, and organizations that want one framework for ETL, analytics and machine learning on clusters.

Apache Hive is data warehouse software built on Apache Hadoop. It makes it possible to read, write and manage very large datasets in distributed storage using SQL, so analysts can run extract-transform-load jobs, reports and data analysis without writing low-level processing code.

Hive provides tools for accessing data through SQL, a way of imposing structure on a variety of data formats, and access to files stored in Apache HDFS or other systems such as Apache HBase. Queries can run on the Apache Tez framework, which is designed for interactive queries and has far lower overhead than MapReduce. It supports much standard SQL functionality, including many analytics features from the 2003 and 2011 SQL standards.

Hive is an Apache Software Foundation project written in Java and licensed under Apache-2.0. You deploy it on your own Hadoop-based clusters, though hosted big data platforms from cloud providers also bundle it. It suits data engineering teams maintaining batch analytics on large distributed data.