Pachyderm

Open source

Pachyderm is an open-source data pipeline and data versioning system on Kubernetes that tracks lineage and reruns pipelines when data changes.

GitHub stars
6.3k
Last 30 days
+0
Last commit
1 yr ago
Repository age
12 years
Version
v2.12.2
Licence
Apache-2.0
Self-hosted
Yes

About Pachyderm

Pachyderm automates data transformations by combining pipelines with data versioning and lineage. Its pipelines are multi-stage and language-agnostic, process data in parallel, and work on any type of data.

Pipelines are data-driven: they trigger automatically when changes in the data are detected. Every change is versioned, giving an immutable record of where data came from. Storage uses standard object stores with automatic deduplication, and autoscaling and parallel processing are built on Kubernetes.

It runs across the major cloud providers and on-premises. The README suggests running it locally or deploying to AWS, GCE or Azure in about five minutes. The code is Go under Apache 2.0, and contributors sign a Contributor License Agreement. The project sends anonymised usage metrics by default.

Key features

  • Pipelines trigger automatically on data changes
  • Immutable lineage and versioning for any data type
  • Autoscaling and parallel processing on Kubernetes
  • Stores data in standard object stores with deduplication
  • Runs on major clouds and on-premises
  • Multi-stage, language-agnostic pipelines

Good fit for

  • Version-controlled data pipelines
  • Tracking data lineage
  • Automating complex data transformations
Tags
data pipelines
data versioning
kubernetes
data lineage
go
data engineering
mlops

Pachyderm: questions and answers

What is Pachyderm used for?
Pachyderm is an open-source data pipeline and data versioning system on Kubernetes that tracks lineage and reruns pipelines when data changes. It is a good fit for version-controlled data pipelines, tracking data lineage and automating complex data transformations.
Is Pachyderm open source?
Yes. Pachyderm is open source under the Apache-2.0 licence. Its source code is on GitHub at pachyderm/pachyderm and is written mainly in Go.
Is Pachyderm free?
Yes. Pachyderm is open source, so the software itself is free to use.
Can I self-host Pachyderm?
Yes. Pachyderm can be self-hosted on your own server or infrastructure.
What is Pachyderm an alternative to?
Pachyderm is an open-source alternative to Databricks, Amazon SageMaker, Arize AI and Weights & Biases. Other open-source alternatives to Amazon SageMaker include Kubeflow, MLflow and Polyaxon.
Is Pachyderm actively maintained?
The most recent commit to Pachyderm was on 3 February 2025, and the latest release is v2.12.2, published on 15 January 2025. The project has 6.3k stars on GitHub.

Open-source alternatives to Pachyderm

See all

SaaS alternatives to Pachyderm

See all