About vLLM
vLLM is an open-source library for running and serving large language models efficiently. It originated at UC Berkeley's Sky Computing Lab and is now developed by a large community of contributors from academia and industry. It is written in Python and released under the Apache-2.0 license.
Speed comes from techniques such as PagedAttention for attention key-value memory, batching incoming requests continuously, chunked prefill, prefix caching, CUDA and HIP graphs, and optimized attention and mixture-of-experts kernels. It supports many quantization formats, including FP8, INT8, INT4, GPTQ and AWQ, as well as speculative decoding and disaggregated prefill and decode.
On the usability side, vLLM integrates with Hugging Face models and supports parallel sampling, beam search, streaming outputs, structured outputs and tool calling, plus several parallelism modes (tensor, pipeline, data, expert and context) for distributed inference. It targets NVIDIA, AMD and TPU hardware and suits teams that serve open models in production and need high throughput.
Key features
- PagedAttention and continuous batching
- Quantization including FP8, INT8, GPTQ and AWQ
- Speculative decoding support
- Tensor, pipeline and expert parallelism
- OpenAI-compatible API server
- Hugging Face model integration
- Streaming and structured outputs
Good fit for
- Serving open LLMs in production
- Maximizing GPU throughput for inference
- Distributed inference across multiple GPUs
vLLM: questions and answers
- What is vLLM used for?
- vLLM is a Python library and server for fast, memory-efficient LLM inference and serving, with PagedAttention, continuous batching and broad quantization support. It is a good fit for serving open LLMs in production, maximizing GPU throughput for inference and distributed inference across multiple GPUs.
- Is vLLM open source?
- Yes. vLLM is open source under the Apache-2.0 licence. Its source code is on GitHub at vllm-project/vllm and is written mainly in Python.
- Is vLLM free?
- Yes. vLLM is open source, so the software itself is free to use.
- Can I self-host vLLM?
- Yes. vLLM can be self-hosted on your own server or infrastructure; there is no official hosted version.
- What is vLLM an alternative to?
- vLLM is an open-source alternative to Amazon Bedrock, Amazon SageMaker, Replicate and Together AI. Other open-source alternatives to Amazon Bedrock include SGLang and Hugging Face Chat UI.
- Is vLLM actively maintained?
- Yes. The most recent commit to vLLM was on 13 November 2026, and the latest release is v0.33.0rc3, published on 2 November 2026. The project has 97k stars on GitHub, with 3.2k added in the last 30 days.
Open-source alternatives to vLLM
See all
SGLang
AI Infrastructure
SGLang is a high-performance serving framework for large language models and multimodal mo
Apache-2.0vs Amazon Bedrock★ 39k
Hugging Face Chat UI
AI Tools
The open source codebase powering HuggingChat
Apache-2.0vs ChatGPT★ 11k
LocalAI
AI Infrastructure
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video -
MITvs ChatGPT★ 50k
Ollama
AI Infrastructure
Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other model
MITvs ChatGPT★ 187k
Ray
Hosting & PaaS
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Li
Apache-2.0vs OpenRouter★ 44k
MLflow
AI Infrastructure
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables te
Apache-2.0vs Weights & Biases★ 29k
SaaS alternatives to vLLM
See all
Amazon Bedrock
AI Infrastructure
Managed AWS service for building generative AI apps with models from several providers
SaaS
Amazon SageMaker
AI Infrastructure
AWS platform for building, training and deploying machine learning models
SaaS
Replicate
AI Infrastructure
API platform for running open-source AI models in the cloud
SaaS
Together AI
AI Infrastructure
Cloud platform for running, fine-tuning and training open and custom AI models
SaaS
Fireworks AI
AI Infrastructure
Inference platform for running and fine-tuning generative AI models at low latency
SaaS
Groq
AI Infrastructure
AI inference cloud and API built on custom LPU hardware for fast model serving
SaaS

Cerebras
SambaNova
Baseten
DeepInfra
Google Vertex AI
Azure AI Foundry
OpenAI API Platform
Cohere
Mistral AI
xAI API
NVIDIA AI Enterprise
Cloudflare Workers AI
OpenRouter