Open-source Valohai alternatives

MLflow is an open-source AI engineering platform for agents, LLM applications and ML models. It helps teams debug, evaluate, monitor and optimize AI applications in production while keeping control of costs and access to models and data. It is written in Python, released under the Apache-2.0 license, and has been developed since 2018, with topics covering MLOps, model management and LLMOps.

For LLM and agent work it offers production observability with traces, plus evaluation, prompt management and prompt optimization, and an AI Gateway that governs costs and model access. It works with Python, TypeScript and JavaScript, Java and other languages, and integrates with OpenTelemetry and MCP. Getting started means starting an MLflow server, enabling logging in your code and running it, then exploring traces and metrics in the web UI on port 5000. A setup wizard can connect to an MLflow server or a Databricks workspace and add tracing with a coding agent.

CML, or Continuous Machine Learning, is an open-source command-line tool for applying CI/CD practices to machine learning work, with a focus on MLOps. It is aimed at data science teams who want to automate workflows such as provisioning machines, training and evaluating models, and comparing experiments across a project's history, using the Git-based tools they already rely on.

It can train and evaluate models automatically on every pull request and generate a visual report of results and metrics directly in that pull request, following a 'GitFlow for data science' approach where GitLab, GitHub or Bitbucket track who changed models or data and when, often paired with DVC to version data and models outside of Git itself. CML needs no additional backend services, databases, or complex setup, working on top of existing GitHub Actions, GitLab CI or Bitbucket Pipelines infrastructure with either self-hosted or cloud runners such as AWS EC2 or Azure.

CML is written in JavaScript and released under the Apache-2.0 license. The project provides a Discord community and a YouTube tutorial series for hands-on MLOps guidance, alongside documentation covering setup for each supported Git platform.

labml is an experiment-tracking and monitoring toolkit aimed at machine learning practitioners who want to check on long-running training jobs, including from their phone, without staying tied to a terminal or a single workstation. It integrates with common deep learning frameworks and tracks experiment metadata alongside training metrics.

With as little as two lines of code, it can track git commit information, configuration values and hyperparameters for an experiment, alongside the usual loss and metric curves, and it works with PyTorch, PyTorch Lightning, Keras, TensorFlow and fastai based on its listed topics. A single command also monitors hardware usage on any machine. An optional self-hosted experiments server, backed by MongoDB and optionally fronted by Nginx, provides the web interface for viewing experiments remotely, including from a mobile browser.

labml is written in Python and released under the MIT license. It is installed via pip, both for the monitoring client embedded in training scripts and for the self-hosted server component, and its documentation covers the Python API, custom visualizations and configuration management in more depth.