Pachyderm

Pachyderm is a data pipeline platform built for data engineering teams that need to automate complex transformations across large and varied datasets. It positions itself as a CI/CD engine for data, combining pipeline automation with built-in version control and lineage so that every change to data and every pipeline run can be traced.

Pipelines trigger automatically when Pachyderm detects a change in input data, and each stage can be written in any language since Pachyderm treats containers as the unit of processing. It tracks immutable data lineage and versioning for any data type, scales and parallelizes processing automatically on top of Kubernetes, and stores data in standard object stores with automatic deduplication. It runs on all major cloud providers as well as on-premises.

Pachyderm is written in Go and released under the Apache-2.0 license. It can be run locally for evaluation or deployed on AWS, GCE or Azure in a short setup, and the project provides tutorials, example projects and case studies through its documentation.