Optimus

Open source

Optimus is a Go workflow orchestrator from Raystack that runs SQL and YAML based data transformation jobs, handling dependencies and scheduling.

raystack.github.io
Optimus homepage screenshot
GitHub stars
770
Last 30 days
+3(+0%)
Last commit
2 yr ago
Repository age
5 years
Version
v0.7.1
Licence
Apache-2.0
Self-hosted
Yes

About Optimus

Optimus is a workflow orchestrator for data transformation, data modeling, pipelines and data quality management. Analysts and engineers write simple SQL queries and YAML configuration, and Optimus takes care of dependency management, scheduling and running transformation jobs at scale. The listing name Raystack is the organization behind it.

Key capabilities include managing warehouse tables and views through YAML, scheduling SQL transformations, automatically parsing queries to build dependency graphs rather than hand-writing DAGs, dry-running queries during deployment, and compile-time templating with variables, loops, conditionals and macros. It also supports cross-tenant dependencies, post-transformation hooks such as sinking BigQuery tables to Kafka, Python transformations and custom plugins.

Specifications are managed in git and the service is driven through REST and gRPC. The topics point to Airflow and BigQuery integrations, and the Go codebase is licensed under Apache-2.0. It suits data teams building ELT or analytics engineering workflows.

Key features

  • SQL and YAML based transformation jobs
  • Automatic dependency graph resolution
  • Warehouse tables and views as YAML
  • Dry runs before deployment
  • Query templating with loops and macros
  • Hooks, Python transformations and plugins

Good fit for

  • ELT pipelines on BigQuery
  • Analytics engineering with SQL
Built with
Go tools
Tags
data-pipelines
orchestration
elt
sql
bigquery
golang
airflow
dataops

Optimus: questions and answers

What is Optimus used for?
Optimus is a Go workflow orchestrator from Raystack that runs SQL and YAML based data transformation jobs, handling dependencies and scheduling. It is a good fit for ELT pipelines on BigQuery and analytics engineering with SQL.
Is Optimus open source?
Yes. Optimus is open source under the Apache-2.0 licence. Its source code is on GitHub at raystack/optimus and is written mainly in Go.
Is Optimus free?
Yes. Optimus is open source, so the software itself is free to use.
Can I self-host Optimus?
Yes. Optimus can be self-hosted on your own server or infrastructure.
What are some alternatives to Optimus?
Similar open-source tools in the Data Pipelines & ETL category include Mage AI, Bruin and Kestra. SaaS products in the same category include Astronomer, Rivery and Azure Data Factory.
Is Optimus actively maintained?
The most recent commit to Optimus was on 8 June 2024, and the latest release is v0.7.1, published on 25 July 2023. The project has 770 stars on GitHub, with 3 added in the last 30 days.

Open-source alternatives to Optimus

See all

SaaS alternatives to Optimus

See all