About Pachyderm
Pachyderm automates data transformations by combining pipelines with data versioning and lineage. Its pipelines are multi-stage and language-agnostic, process data in parallel, and work on any type of data.
Pipelines are data-driven: they trigger automatically when changes in the data are detected. Every change is versioned, giving an immutable record of where data came from. Storage uses standard object stores with automatic deduplication, and autoscaling and parallel processing are built on Kubernetes.
It runs across the major cloud providers and on-premises. The README suggests running it locally or deploying to AWS, GCE or Azure in about five minutes. The code is Go under Apache 2.0, and contributors sign a Contributor License Agreement. The project sends anonymised usage metrics by default.
Key features
- Pipelines trigger automatically on data changes
- Immutable lineage and versioning for any data type
- Autoscaling and parallel processing on Kubernetes
- Stores data in standard object stores with deduplication
- Runs on major clouds and on-premises
- Multi-stage, language-agnostic pipelines
Good fit for
- Version-controlled data pipelines
- Tracking data lineage
- Automating complex data transformations
Pachyderm: questions and answers
- What is Pachyderm used for?
- Pachyderm is an open-source data pipeline and data versioning system on Kubernetes that tracks lineage and reruns pipelines when data changes. It is a good fit for version-controlled data pipelines, tracking data lineage and automating complex data transformations.
- Is Pachyderm open source?
- Yes. Pachyderm is open source under the Apache-2.0 licence. Its source code is on GitHub at pachyderm/pachyderm and is written mainly in Go.
- Is Pachyderm free?
- Yes. Pachyderm is open source, so the software itself is free to use.
- Can I self-host Pachyderm?
- Yes. Pachyderm can be self-hosted on your own server or infrastructure.
- What is Pachyderm an alternative to?
- Pachyderm is an open-source alternative to Databricks, Amazon SageMaker, Arize AI and Weights & Biases. Other open-source alternatives to Amazon SageMaker include Kubeflow, MLflow and Polyaxon.
- Is Pachyderm actively maintained?
- The most recent commit to Pachyderm was on 3 February 2025, and the latest release is v2.12.2, published on 15 January 2025. The project has 6.3k stars on GitHub.
Open-source alternatives to Pachyderm
See all
Kubeflow
AI Tools
The Cloud Native AI Platform
Apache-2.0vs Together AI★ 16k
MLflow
AI Infrastructure
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables te
Apache-2.0vs Weights & Biases★ 29k
Dvc
AI Tools
🦉 Data Versioning and ML Experiments
Apache-2.0vs Arize AI★ 16k
Polyaxon
Workflow Automation
AI Infra / AI Orchestration / AI Control Plane
Apache-2.0vs Amazon SageMaker★ 3.7k
BentoML
AI Infrastructure
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM
Apache-2.0vs Amazon SageMaker★ 8.9k
Flyte
AI Tools
Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on y
Apache-2.0vs Amazon SageMaker★ 7.8k
SaaS alternatives to Pachyderm
See all
Databricks
Data Pipelines & ETL
Lakehouse platform for data engineering, analytics and machine learning
SaaS
Amazon SageMaker
AI Infrastructure
AWS platform for building, training and deploying machine learning models
SaaS
Arize AI
AI Infrastructure
Observability and evaluation platform for machine learning models and LLM applications
SaaS
Weights & Biases
AI Infrastructure
Experiment tracking, model registry and LLM evaluation for machine learning teams
SaaS
Domino Data Lab
AI Infrastructure
Enterprise MLOps platform for building, governing and deploying data science models
SaaS
NVIDIA AI Enterprise
AI Infrastructure
Software suite of NIM microservices, frameworks and tools for running production AI
SaaS

Valohai