promptfoo

Promptfoo is a CLI and library designed for evaluating and red-teaming Large Language Model (LLM) applications. It helps developers build secure and reliable AI apps, moving beyond trial-and-error. It offers automated evaluations, vulnerability scanning, side-by-side model comparison, and CI/CD integration. A key feature is its ability to run evaluations 100% locally, ensuring privacy. It is developer-first, flexible, battle-tested, and provides data-driven insights.

• Automated LLM evaluations and benchmarking
• LLM app red teaming and vulnerability scanning
• Side-by-side comparison of multiple models (OpenAI, Anthropic, Azure, etc.)
• CI/CD automation and integration
• Local and private execution for data security