Open-source Google Vertex AI alternatives

vLLM is an open-source library for running and serving large language models efficiently. It originated at UC Berkeley's Sky Computing Lab and is now developed by a large community of contributors from academia and industry. It is written in Python and released under the Apache-2.0 license.

Speed comes from techniques such as PagedAttention for attention key-value memory, batching incoming requests continuously, chunked prefill, prefix caching, CUDA and HIP graphs, and optimized attention and mixture-of-experts kernels. It supports many quantization formats, including FP8, INT8, INT4, GPTQ and AWQ, as well as speculative decoding and disaggregated prefill and decode.

On the usability side, vLLM integrates with Hugging Face models and supports parallel sampling, beam search, streaming outputs, structured outputs and tool calling, plus several parallelism modes (tensor, pipeline, data, expert and context) for distributed inference. It targets NVIDIA, AMD and TPU hardware and suits teams that serve open models in production and need high throughput.

SGLang is an inference framework, released as open source, for language models, vision-language models and diffusion models. It is optimized for agentic workloads, reinforcement-learning rollouts and large-scale serving. The project includes SGLang Diffusion, a built-in engine for image and video generation that ships in the same repository and Python package. It is released under the Apache-2.0 license.

Getting started takes either a prebuilt Docker image or a Python install using uv, followed by a launch command for the chosen model. A cookbook helps pick a model and hardware pair and generates a ready-to-run command. Supported hardware spans NVIDIA and AMD GPUs, Google TPUs, Intel GPUs and CPUs, Apple Silicon through Metal and MLX, and Huawei Ascend NPUs, with more integrations in progress. The wider SGLang ecosystem adds educational projects and community events.

MLflow is an open-source AI engineering platform for agents, LLM applications and ML models. It helps teams debug, evaluate, monitor and optimize AI applications in production while keeping control of costs and access to models and data. It is written in Python, released under the Apache-2.0 license, and has been developed since 2018, with topics covering MLOps, model management and LLMOps.

For LLM and agent work it offers production observability with traces, plus evaluation, prompt management and prompt optimization, and an AI Gateway that governs costs and model access. It works with Python, TypeScript and JavaScript, Java and other languages, and integrates with OpenTelemetry and MCP. Getting started means starting an MLflow server, enabling logging in your code and running it, then exploring traces and metrics in the web UI on port 5000. A setup wizard can connect to an MLflow server or a Databricks workspace and add tracing with a coding agent.

Backend.AI, from Lablup, is a streamlined, container-based computing cluster platform that hosts popular machine learning frameworks and many programming languages. Its distinguishing feature is pluggable support for heterogeneous accelerators, including CUDA and ROCm GPUs, Intel Gaudi, Google TPU, Graphcore IPU and NPUs from vendors such as Rebellions, FuriosaAI, HyperAccel and Tenstorrent.

It allocates and isolates computing resources for multi-tenant sessions on demand or in batches, using customizable job schedulers and its own orchestrator named Sokovan. All functions are exposed through REST and GraphQL APIs. The README lists requirements including Python 3.13 with Pantsbuild, Docker 20.10 or newer or Podman 5.4 or newer, Docker Compose v2, PostgreSQL 16 or newer, Valkey (Redis-compatible), etcd and Prometheus, with Grafana recommended for observability.

Backend.AI is written in Python and licensed under LGPL-3.0. It is aimed at organizations running shared GPU or accelerator clusters for research and ML workloads, and you operate it on your own infrastructure.

KubeDL is a Kubernetes-based system that makes it easier and more efficient to run deep learning workloads. It is a CNCF sandbox project and handles both training and inference jobs for frameworks such as TensorFlow, PyTorch and Mars through a single unified controller.

Features include advanced scheduling, cache-based acceleration, metadata persistence, file synchronization and service discovery for training jobs in host networking. It can automatically tune the best configuration for ML model deployment through the related Morphling project, and it can package and deploy models in containers while tracking model lineage natively using Kubernetes custom resources.

KubeDL is written in Go and released under Apache-2.0. A related research paper on Morphling, an auto-configuration approach for cloud-native model serving, is cited in the README. It suits platform teams that run machine learning jobs on shared Kubernetes clusters.