Open-source Amazon Bedrock alternatives

Dify is a platform for building applications on top of large language models. Instead of wiring together prompts, retrieval code and model APIs by hand, teams assemble AI workflows, knowledge pipelines and agents in a shared visual workspace, then move them from prototype to production without changing stacks.

The product covers a workflow studio for agentic flows, a knowledge pipeline that prepares searchable knowledge bases for retrieval-augmented generation, model management across many providers, and a marketplace of tools, models and integrations. Observability integrations such as Opik, Langfuse and Arize Phoenix help trace and debug runs once an app is live.

Dify comes in a few editions: a community edition you deploy yourself with Docker, a hosted cloud service, and an enterprise edition for private deployments. The repository carries a custom license rather than a standard one, so check its terms before offering Dify as a service to others.

vLLM is an open-source library for running and serving large language models efficiently. It originated at UC Berkeley's Sky Computing Lab and is now developed by a large community of contributors from academia and industry. It is written in Python and released under the Apache-2.0 license.

Speed comes from techniques such as PagedAttention for attention key-value memory, batching incoming requests continuously, chunked prefill, prefix caching, CUDA and HIP graphs, and optimized attention and mixture-of-experts kernels. It supports many quantization formats, including FP8, INT8, INT4, GPTQ and AWQ, as well as speculative decoding and disaggregated prefill and decode.

On the usability side, vLLM integrates with Hugging Face models and supports parallel sampling, beam search, streaming outputs, structured outputs and tool calling, plus several parallelism modes (tensor, pipeline, data, expert and context) for distributed inference. It targets NVIDIA, AMD and TPU hardware and suits teams that serve open models in production and need high throughput.

LiteLLM is an AI gateway, open source, offering one unified interface for calling more than 100 LLM providers, including OpenAI, Anthropic, Gemini, Bedrock and Azure, using the OpenAI request format. It can be used as a Python SDK inside an application or deployed as a proxy server that serves a whole team or organization. The core is Python with a Rust component, and GitHub lists the license as 'Other'.

The project aims to remove the hassle of handling a separate SDK, login scheme, request format and set of errors for each model. On top of that the gateway provides load balancing, guardrails, spend tracking, virtual keys and an admin dashboard, and it exposes endpoints for chat completions, responses, embeddings, images, audio, batches and reranking.

Because requests keep the OpenAI format, teams can swap providers without rewriting application code. It is self-hosted, with hosted-proxy and enterprise tier options linked in the README, and its topics also reference an MCP gateway and LLMOps. It suits platform and AI engineering teams that need central control over model access, budgets and logging.

SGLang is an inference framework, released as open source, for language models, vision-language models and diffusion models. It is optimized for agentic workloads, reinforcement-learning rollouts and large-scale serving. The project includes SGLang Diffusion, a built-in engine for image and video generation that ships in the same repository and Python package. It is released under the Apache-2.0 license.

Getting started takes either a prebuilt Docker image or a Python install using uv, followed by a launch command for the chosen model. A cookbook helps pick a model and hardware pair and generates a ready-to-run command. Supported hardware spans NVIDIA and AMD GPUs, Google TPUs, Intel GPUs and CPUs, Apple Silicon through Metal and MLX, and Huawei Ascend NPUs, with more integrations in progress. The wider SGLang ecosystem adds educational projects and community events.

Gateway is Portkey's open-source AI gateway, a lightweight service for routing requests to a large catalog of language, vision, audio and image models through one API. It is designed to be fast, reliable and enterprise-ready, and the project says you can integrate with any language model quickly. It also bundles AI guardrails and, per its topics, supports MCP gateway use. It is written in TypeScript under the MIT license.

A quickstart runs the gateway locally, serving the API on port 8787 along with a console. Deployment guides cover Portkey Cloud, Docker, Node.js, Cloudflare, Replit and other targets. Libraries and SDKs listed include JavaScript, Python, REST, the OpenAI SDKs, LangChain and LlamaIndex, so existing applications can point at the gateway with small changes.

The README announces a Gateway 2.0 pre-release in which Portkey's core enterprise gateway merges into open source, and it mentions an open-source model pricing dataset called Portkey Models. A hosted gateway and enterprise offering exist from the vendor, so the project suits both teams who want to self-host routing and those who prefer a managed service.

LLM Gateway is an open-source API gateway for large language models. It sits between your applications and multiple providers such as OpenAI, Anthropic and Google Vertex AI, so your code talks to one interface while the gateway decides where each request goes. Its API follows the OpenAI format, which keeps migration from existing code straightforward.

Beyond routing, it centralizes the management of provider API keys and tracks requests, token usage, response times and costs so teams can see what their LLM usage is doing. Performance monitoring lets you compare models on speed and cost-effectiveness. Topics on the repository also mention guardrails, rate limiting, observability and enterprise use.

You can use LLM Gateway in two ways: a hosted version at llmgateway.io where you create an account and get an API key, or a self-hosted deployment on your own infrastructure using a unified Docker image with PostgreSQL. The code is written in TypeScript and its licence is listed as other, so check the repository terms. It suits developers and platform teams who juggle several model vendors.