About Promptfoo
Promptfoo is an open-source command-line tool and library for testing applications built on large language models. Developers describe test cases and checks in declarative configuration files, run them against one or more models, and review the results, which replaces manual trial and error with repeatable evaluations. It is written in TypeScript and released under the MIT license.
Beyond quality testing, Promptfoo includes red teaming and vulnerability scanning for AI systems, probing prompts, agents, and retrieval-augmented generation (RAG) pipelines for security and safety weaknesses. It can compare models from providers such as OpenAI, Anthropic, Azure, Bedrock, and Ollama side by side, and it works with any LLM API or programming language. Its documentation also covers a code-scanning feature that reviews pull requests for LLM-related security and compliance issues.
The project is designed to run locally: the maintainers state that evaluations execute on the developer's own machine, so prompts only leave it when a chosen model provider receives them. It can be installed with Homebrew or pip, or run without installing via npx, and it plugs into CI/CD so checks run automatically on each change. The README notes that Promptfoo is now part of OpenAI while remaining open source and MIT licensed.
Key features
- Declarative test configs for prompts and models
- Side-by-side comparison of multiple LLM providers
- Red teaming and vulnerability scanning for AI apps
- Evaluations run locally on your own machine
- CI/CD integration for automated checks
- Pull request code scanning for LLM risks
- Live reload and result caching
- Shareable results and security reports
Good fit for
- Regression testing prompts before release
- Comparing models for a specific task
- Security testing of chatbots and agents
- Gating LLM changes in CI pipelines
Promptfoo: questions and answers
- What is Promptfoo used for?
- Promptfoo is an open-source CLI and library for evaluating prompts, agents, and RAG apps, and for red-teaming LLM applications to find security weaknesses. It is a good fit for regression testing prompts before release, comparing models for a specific task, and security testing of chatbots and agents.
- Is Promptfoo open source?
- Yes. Promptfoo is open source under the MIT licence. Its source code is on GitHub at promptfoo/promptfoo and is written mainly in TypeScript.
- Is Promptfoo free?
- Yes. Promptfoo is open source, so the software itself is free to use.
- What is Promptfoo an alternative to?
- Promptfoo is an open-source alternative to Braintrust, Galileo and Vellum. Other open-source alternatives to Braintrust include Langfuse, Arize Phoenix and Opik.
- Is Promptfoo actively maintained?
- Yes. The most recent commit to Promptfoo was on 12 November 2026, and the latest release is 0.125.1, published on 25 October 2026. The project has 27k stars on GitHub, with 1.2k added in the last 30 days.
Open-source alternatives to Promptfoo
See all
Langfuse
AI Infrastructure
Trace, evaluate, and improve AI agents with one open platform. Use production data to unde
OSSvs Weights & Biases★ 37k
Arize Phoenix
AI Infrastructure
AI Observability & Evaluation
OSSvs Weights & Biases★ 12k
Opik
AI Infrastructure
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows wit
Apache-2.0vs Weights & Biases★ 23k
Laminar
AI Infrastructure
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
Apache-2.0vs Weights & Biases★ 3.5k
LangWatch
AI Infrastructure
The platform for LLM evaluations and AI agent testing
Apache-2.0vs LangSmith★ 5k
Helicone
AI Infrastructure
🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and expe
Apache-2.0vs OpenRouter★ 6.3k
SaaS alternatives to Promptfoo
See all
Braintrust
AI Infrastructure
Evaluation and observability platform for building and testing AI applications
SaaS
Galileo
AI Infrastructure
Evaluation and monitoring platform for generative AI applications
SaaS
Vellum
AI Infrastructure
Platform for building, testing and monitoring LLM prompts and workflows
SaaS
LangSmith
AI Infrastructure
Observability, tracing and evaluation platform for LLM applications
SaaS
Arize AI
AI Infrastructure
Observability and evaluation platform for machine learning models and LLM applications
SaaS
PromptVault
AI Infrastructure
System prompt CMS for LLM builders so you can fix a prompt without redeploying your app
SaaS

