https://github.com/promptfoo/promptfoo

ci ci-cd cicd evaluation evaluation-framework llm llm-eval llm-evaluation llm-evaluation-framework llmops pentesting prompt-engineering prompt-testing prompts rag red-teaming testing vulnerability-scanners

Score: 28.64760004826949

Last synced: about 10 hours ago
JSON representation

Repository metadata:

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Host: GitHub
URL: https://github.com/promptfoo/promptfoo
Owner: promptfoo
License: mit
Created: 2023-04-28T15:48:49.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2026-06-07T19:24:52.000Z (6 days ago)
Last Synced: 2026-06-07T22:07:40.258Z (6 days ago)
Topics: ci, ci-cd, cicd, evaluation, evaluation-framework, llm, llm-eval, llm-evaluation, llm-evaluation-framework, llmops, pentesting, prompt-engineering, prompt-testing, prompts, rag, red-teaming, testing, vulnerability-scanners
Language: TypeScript
Homepage: https://promptfoo.dev
Size: 619 MB
Stars: 21,987
Watchers: 57
Forks: 1,947
Open Issues: 318
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
- Agents: .github/AGENTS.md
Funding:
- Github: typpo

Owner metadata:

Name: promptfoo
Login: promptfoo
Email:
Kind: organization
Description: Test your prompts
Website: https://www.promptfoo.dev/
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/137907881?v=4
Repositories: 2
Last Synced at: 2023-06-28T16:35:52.573Z
Profile URL: https://github.com/promptfoo

Committers metadata

Last synced: 3 days ago

Total Commits: 8,907
Total Committers: 298
Avg Commits per committer: 29.889
Development Distribution Score (DDS): 0.674

Commits in past year: 4,267
Committers in past year: 152
Avg Commits per committer in past year: 28.072
Development Distribution Score (DDS) in past year: 0.66

Name	Email	Commits
ianw_github	i**b@i**m	2904
mldangelo	7****o	2578
renovate[bot]	2****]	890
dependabot[bot]	4****]	447
sklein12	s**e@p**v	321
will-holley	8****y	217
MrFlounder	k**s@g**m	190
faizanminhas	5****s	183
gru-agent[bot]	1****]	105
jbeckwith	j**h@g**m	84
jameshiester	5****r	59
kkahadze	8****e	46
promptfoobot[bot]	2****]	41
danenania	d**e@p**v	40
yash2998chhabria	y**a@g**m	39
AI Simply Explained	a**9@g**m	38
vedantr	v****r	36
use-tusk[bot]	1****]	34
vsauter	4****r	29
addelong	a**n@p**v	28
Ryanba	9****h	25
abrayne	4****e	20
minhle1291	m**1@g**m	19
devin-ai-integration[bot]	1****]	15
Tabs	t**b@l**m	14
Swarna Podila	1****p	11
Evan Bonsignori	e**i@g**m	11
github-actions[bot]	4****]	9
Albert Lie 이영덕	a**3@g**m	8
Sebastien Stettler	s**r@g**m	8
and 268 more...

Issue and Pull Request metadata

Last synced: 3 days ago

Total issues: 765
Total pull requests: 4,040
Average time to close issues: 16 days
Average time to close pull requests: 2 days
Total issue authors: 394
Total pull request authors: 210
Average comments per issue: 1.64
Average comments per pull request: 0.98
Merged pull request: 2,937
Bot issues: 5
Bot pull requests: 745

Past year issues: 89
Past year pull requests: 984
Past year average time to close issues: 6 days
Past year average time to close pull requests: 2 days
Past year issue authors: 66
Past year pull request authors: 54
Past year average comments per issue: 0.89
Past year average comments per pull request: 1.63
Past year merged pull request: 567
Past year bot issues: 1
Past year bot pull requests: 177

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/promptfoo/promptfoo

Top Issue Authors

jamesbraza (18)
albertlieyingadrian (17)
aantn (15)
mldangelo (13)
typpo (12)
SysOverdrive (10)
efung (10)
pelikhan (10)
romaintoub (8)
rvernica (8)
chrismaltais (8)
zhlmmc (8)
sbichenko (8)
billybonks (7)
AmineDjeghri (7)

Top Pull Request Authors

mldangelo (1,184)
typpo (1,025)
gru-agent[bot] (392)
dependabot[bot] (300)
sklein12 (251)
will-holley (142)
MrFlounder (114)
faizanminhas (84)
AISimplyExplained (41)
vedantr (37)
devin-ai-integration[bot] (28)
vsauter (27)
abrayne (21)
use-tusk[bot] (18)
mldangelo-oai (10)

Top Issue Labels

bug (31)
enhancement (30)
question (25)
good first issue (6)
dependencies (6)
javascript (5)
in-progress (5)
Open Source (4)
documentation (3)
wontfix (2)
help wanted (2)
codex (1)

Top Pull Request Labels

dependencies (300)
javascript (277)
codex (62)
python (16)
bug (6)
github_actions (5)
enhancement (4)
codex-automation (4)
sourcery (3)
documentation (2)
autorelease: pending (2)
question (1)
in-progress (1)
good first issue (1)

Package metadata

Total packages: 4
Total downloads:
- npm: 415,785 last-month
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 1 (may contain duplicates)
Total versions: 433
Total maintainers: 6

proxy.golang.org: github.com/promptfoo/promptfoo/examples/golang-provider

Package main implements a promptfoo provider that uses OpenAI's API. It demonstrates a simple implementation of the provider interface using shared code from the core and pkg1 packages.

Homepage: https://github.com/promptfoo/promptfoo
Documentation: https://pkg.go.dev/github.com/promptfoo/promptfoo/examples/golang-provider#section-documentation
Licenses: MIT
Latest release: v0.0.0-20260302231834-056e9c0640a5 (published 3 months ago)
Last Synced: 2026-06-01T19:02:13.375Z (12 days ago)
Versions: 20
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent packages count: 5.704%
- Average: 5.896%
- Dependent repos count: 6.087%

proxy.golang.org: github.com/promptfoo/promptfoo

Homepage: https://github.com/promptfoo/promptfoo
Documentation: https://pkg.go.dev/github.com/promptfoo/promptfoo#section-documentation
Licenses: MIT
Latest release: v0.103.14 (published over 1 year ago)
Last Synced: 2026-06-09T23:02:28.289Z (3 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent packages count: 6.194%
- Average: 6.402%
- Dependent repos count: 6.609%

npmjs.org: promptfoo

LLM eval & testing toolkit

Homepage: https://promptfoo.dev
Licenses: MIT
Latest release: 0.121.13 (published 15 days ago)
Last Synced: 2026-06-01T19:02:26.121Z (12 days ago)
Versions: 411
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 415,776 Last month
Rankings:
- Downloads: 0.804%
- Stargazers count: 2.307%
- Forks count: 4.269%
- Dependent repos count: 10.302%
- Average: 13.908%
- Dependent packages count: 51.857%
Maintainers (5)

npmjs.org: @alexcarol/promptfoo

LLM eval & testing toolkit

Homepage: https://github.com/promptfoo/promptfoo#readme
Status: removed
Licenses: MIT
Latest release: 0.119.11 (published 7 months ago)
Last Synced: 2026-06-09T23:02:28.758Z (3 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 9 Last month
Rankings:
- Dependent repos count: 23.384%
- Average: 28.555%
- Dependent packages count: 33.727%
Maintainers (1)
- alexcarol

Dependencies

.github/workflows/main.yml actions

actions/checkout v3 composite
actions/setup-node v3 composite
bahmutov/npm-install v1 composite

examples/jest-integration/package.json npm

@types/jest ^29.5.1 development
jest ^29.5.0 development
ts-jest ^29.1.0 development
typescript ^5.0.4 development

examples/node-package/package.json npm

package-lock.json npm

495 dependencies

package.json npm

@types/async ^3.2.20 development
@types/cache-manager ^4.0.2 development
@types/cache-manager-fs-hash ^0.0.1 development
@types/cli-progress ^3.11.0 development
@types/cors ^2.8.13 development
@types/debounce ^1.2.1 development
@types/express ^4.17.17 development
@types/glob ^8.1.0 development
@types/jest ^29.5.1 development
@types/js-yaml ^4.0.5 development
@types/node-fetch ^2.6.4 development
@types/nunjucks ^3.2.2 development
@types/opener ^1.4.0 development
@types/semver ^7.5.0 development
babel-jest ^29.5.0 development
jest ^29.5.0 development
jest-watch-typeahead ^2.2.2 development
prettier ^2.8.8 development
ts-jest ^29.1.0 development
ts-node ^10.9.1 development
typescript ^5.0.4 development
@anthropic-ai/sdk ^0.5.2
@apidevtools/json-schema-ref-parser ^10.1.0
async ^3.2.4
cache-manager ^4.1.0
cache-manager-fs-hash ^1.0.0
chalk ^4.1.2
cli-progress ^3.12.0
cli-table3 ^0.6.3
commander ^10.0.1
cors ^2.8.5
csv-parse ^5.3.8
csv-stringify ^6.3.2
debounce ^1.2.1
express ^4.18.2
glob ^10.2.6
js-yaml ^4.1.0
node-fetch ^2.6.7
nunjucks ^3.2.4
opener ^1.5.2
replicate ^0.12.3
rouge ^1.0.3
semver ^7.5.3
socket.io ^4.6.1
tiny-invariant ^1.3.1
winston ^3.8.2

examples/langchain-python/requirements.txt pypi

PyYAML ==6.0
SQLAlchemy ==2.0.18
aiohttp ==3.8.5
aiosignal ==1.3.1
async-timeout ==4.0.2
attrs ==23.1.0
certifi ==2023.5.7
charset-normalizer ==3.1.0
dataclasses-json ==0.5.9
frozenlist ==1.3.3
greenlet ==2.0.2
idna ==3.4
langchain ==0.0.228
langchainplus-sdk ==0.0.20
marshmallow ==3.19.0
marshmallow-enum ==1.5.1
multidict ==6.0.4
mypy-extensions ==1.0.0
numexpr ==2.8.4
numpy ==1.25.0
openai ==0.27.8
openapi-schema-pydantic ==1.2.4
packaging ==23.1
pydantic ==1.10.11
requests ==2.31.0
tenacity ==8.2.2
tqdm ==4.65.0
typing-inspect ==0.9.0
typing_extensions ==4.7.1
urllib3 ==2.0.3
yarl ==1.9.2

Dockerfile docker

node 16-alpine build

examples/openai-agents-basic/package.json npm

@openai/agents ^0.3.3
zod ^3.25.76

src/app/package.json npm

@emotion/react ^11.14.0 development
@emotion/styled ^11.14.1 development
@kurkle/color ^0.4.0 development
@mui/icons-material ^7.3.5 development
@mui/material ^7.3.5 development
@mui/system ^7.3.5 development
@mui/x-charts ^8.19.0 development
@mui/x-data-grid ^8.19.0 development
@tanstack/react-query ^5.90.10 development
@tanstack/react-table ^8.21.3 development
@tanstack/table-core ^8.21.3 development
@testing-library/dom ^10.4.1 development
@testing-library/jest-dom ^6.9.1 development
@testing-library/react ^16.3.0 development
@testing-library/user-event ^14.6.1 development
@types/js-yaml ^4.0.9 development
@types/node ^24.10.1 development
@types/prismjs ^1.26.5 development
@types/react 19.2.7 development
@types/react-dom 19.2.3 development
@vitejs/plugin-react ^5.1.1 development
@vitest/coverage-v8 ^4.0.14 development
ansi-to-html ^0.7.2 development
chart.js ^4.5.1 development
csv-parse ^6.1.0 development
csv-stringify ^6.6.0 development
diff ^8.0.2 development
idb-keyval ^6.2.2 development
js-yaml ^4.1.1 development
jsdom ^26.1.0 development
posthog-js ^1.298.0 development
prismjs ^1.30.0 development
react 19.2.0 development
react-dom 19.2.0 development
react-error-boundary ^6.0.0 development
react-markdown ^10.1.0 development
react-router-dom ^7.9.6 development
react-simple-code-editor ^0.14.1 development
recharts ^3.5.0 development
remark-gfm ^4.0.1 development
socket.io-client ^4.8.1 development
typescript ^5.9.3 development
use-debounce ^10.0.6 development
vite ^7.2.4 development
vite-plugin-node-polyfills ^0.24.0 development
vitest ^4.0.14 development
zustand ^5.0.8 development

examples/docker-code-generation-sandbox/requirements.txt pypi

certifi ==2025.10.5
charset-normalizer ==3.3.2
docker ==7.1.0
epicbox ==1.1.0
idna ==3.7
python-dateutil ==2.9.0.post0
requests ==2.32.5
six ==1.16.0
structlog ==24.2.0
urllib3 ==2.5.0

examples/redteam-langchain/requirements.txt pypi

PyYAML ==6.0.3
SQLAlchemy ==2.0.43
aiohappyeyeballs ==2.6.1
aiohttp ==3.13.0
aiosignal ==1.4.0
annotated-types ==0.7.0
anyio ==4.11.0
attrs ==25.4.0
certifi ==2025.10.5
charset-normalizer ==3.4.3
dataclasses-json ==0.6.7
distro ==1.9.0
frozenlist ==1.8.0
h11 ==0.16.0
httpcore ==1.0.9
httpx ==0.28.1
httpx-sse ==0.4.2
idna ==3.10
jiter ==0.11.0
jsonpatch ==1.33
jsonpointer ==3.0.0
langchain ==0.3.27
langchain-community ==0.3.31
langchain-core ==0.3.80
langchain-openai ==0.3.35
langchain-text-splitters ==0.3.11
langsmith ==0.4.33
marshmallow ==3.26.1
multidict ==6.7.0
mypy_extensions ==1.1.0
numpy ==2.3.3
openai ==2.2.0
orjson ==3.11.3
packaging ==25.0
propcache ==0.4.0
pydantic ==2.12.0
pydantic-settings ==2.11.0
pydantic_core ==2.41.1
python-dotenv ==1.1.1
regex ==2025.9.18
requests ==2.32.5
requests-toolbelt ==1.0.0
sniffio ==1.3.1
tenacity ==9.1.2
tiktoken ==0.12.0
tqdm ==4.67.1
typing-inspect ==0.9.0
typing-inspection ==0.4.2
typing_extensions ==4.15.0
urllib3 ==2.5.0
yarl ==1.22.0
zstandard ==0.25.0

code-scan-action/package.json npm

@types/jest ^30.0.0 development
@types/node ^24.10.1 development
@vercel/ncc ^0.38.4 development
typescript ^5.9.3 development
@actions/core ^1.11.1
@actions/exec ^1.1.1
@actions/github ^6.0.1
@octokit/auth-app ^8.1.2
@octokit/rest ^22.0.1

examples/golang-provider/go.sum go

github.com/sashabaranov/go-openai v1.37.0

examples/python-provider/requirements.txt pypi

annotated-types ==0.6.0
anyio ==4.2.0
certifi ==2025.10.5
distro ==1.9.0
exceptiongroup ==1.2.0
h11 ==0.16.0
httpcore ==1.0.2
httpx ==0.26.0
idna ==3.7
openai ==2.3.0

examples/rag-full/requirements.txt pypi

chromadb ==1.0.15
langchain ==0.3.27
langchain-chroma ==0.2.4
langchain-community ==0.3.31
langchain-openai ==0.3.35
pypdf ==6.4.0
tqdm ==4.67.1

.github/workflows/docker.yml actions

actions/checkout v5 composite
actions/download-artifact v6 composite
actions/setup-node v6 composite
actions/upload-artifact v5 composite
docker/build-push-action v6 composite
docker/login-action v3 composite
docker/metadata-action v5 composite
docker/setup-buildx-action v3 composite
docker/setup-qemu-action v3 composite

.github/workflows/image-actions.yml actions

actions/checkout v5 composite
calibreapp/image-actions main composite

src/golang/go.mod go

examples/redteam-tracing-example/package.json npm

express ^5.2.1

examples/stateful-session-management/package.json npm

express ^5.2.1
openai ^4.104.0
uuid ^9.0.1

examples/websockets/test-server/package.json npm

ws ^8.18.3

examples/openai-agents/requirements.txt pypi

openai-agents ==0.1.0
pydantic ==2.12.0

.github/workflows/deploy-launcher.yml actions

actions/checkout v5 composite
actions/setup-node v6 composite
cloudflare/wrangler-action v3 composite

examples/strands-agents/requirements.txt pypi

pydantic >=2.0.0
strands-agents >=1.0.0,<2.0.0

examples/http-provider-auth-signature-pfx/package.json npm

express ^5.1.0
pem ^1.15.1
promptfoo ^0.119.12

examples/redteam-medical-agent/package.json npm

cors ^2.8.5
dotenv ^17.2.3
express ^5.1.0
openai ^5.23.2

examples/simple-mcp/package.json npm

promptfoo * development
@modelcontextprotocol/sdk ^1.22.0
zod ^3.25.76

examples/image-classification/requirements.txt pypi

numpy ==1.26.4
pandas ==2.3.3
pillow ==11.3.0
tensorflow ==2.20.0

.github/workflows/tusk-test-runner-vitest-unit-tests.yml actions

Use-Tusk/test-runner v1 composite
actions/checkout v5 composite
actions/setup-node v6 composite

.github/workflows/validate-pr-title.yml actions

amannn/action-semantic-pull-request v6 composite

examples/headless-browser/requirements.txt pypi

gradio >=5.49.1
playwright >=1.55.0

.github/workflows/promptfoo-code-scan.yml actions

actions/checkout v4 composite
promptfoo/code-scan-action v0 composite

.devcontainer/docker-compose.yml docker

examples/node-module-package/node_modules/testpackage/package.json npm

site/package.json npm

@calcom/embed-react ^1.5.3 development
@docusaurus/core ^3.9.2 development
@docusaurus/module-type-aliases ^3.9.2 development
@docusaurus/plugin-client-redirects ^3.9.2 development
@docusaurus/plugin-content-docs ^3.9.2 development
@docusaurus/preset-classic ^3.9.2 development
@docusaurus/theme-mermaid ^3.9.2 development
@mdx-js/react ^3.1.1 development
@monaco-editor/react ^4.7.0 development
@mui/icons-material ^7.3.5 development
@segment/ajv-human-errors ^2.15.0 development
@swc/core ^1.15.3 development
@swc/jest ^0.2.39 development
@tsconfig/docusaurus ^2.0.7 development
canvas-confetti ^1.9.4 development
clsx ^2.1.1 development
dedent ^1.7.0 development
docusaurus-plugin-image-zoom ^3.0.1 development
framer-motion ^12.23.24 development
gray-matter ^4.0.3 development
lucide-react ^0.555.0 development
prism-react-renderer ^2.4.1 development
react 19.2.0 development
react-dom 19.2.0 development
satori ^0.18.3 development
sharp ^0.34.5 development
typescript ^5.9.3 development
@mui/material ^7.3.5
@mui/system ^7.3.5
@types/react 19.2.7
@types/react-dom 19.2.3
react-countup ^6.5.3
swiper ^12.0.0

examples/pydantic-ai/requirements.txt pypi

openai ==2.3.0
pydantic ==2.12.0
pydantic-ai ==1.0.17
python-dotenv ==1.1.1

examples/crewai/requirements.txt pypi

crewai >=0.203.0

examples/http-provider-auth-signature/package.json npm

express ^5.1.0
promptfoo ^0.119.12

examples/node-package-typescript/package.json npm

@types/node ^24.10.1 development
tsx ^4.20.6 development
typescript ^5.9.3 development
promptfoo file:../promptfoo

examples/opentelemetry-tracing/package.json npm

@types/node ^24.10.1 development
typescript ^5.9.3 development
@opentelemetry/api ^1.9.0
@opentelemetry/exporter-trace-otlp-http ^0.208.0
@opentelemetry/resources ^2.2.0
@opentelemetry/sdk-trace-base ^2.2.0
@opentelemetry/sdk-trace-node ^2.2.0
@opentelemetry/semantic-conventions ^1.38.0

examples/redteam-chatbot/package.json npm

express ^5.1.0
promptfoo ^0.119.12

examples/redteam-mcp-agent/package.json npm

@faker-js/faker ^9.9.0
@modelcontextprotocol/sdk ^1.22.0
dotenv ^17.2.3
openai ^5.23.2

examples/bert-score/requirements.txt pypi

bert-score >=0.3.13
torch >=1.0.0
transformers >=4.0.0

.github/workflows/release-please.yml actions

actions/checkout v5 composite
actions/checkout v4 composite
actions/setup-node v6 composite
googleapis/release-please-action v4 composite

.github/workflows/tusk-test-runner-app-vitest-unit-tests.yml actions

Use-Tusk/test-runner v1 composite
actions/checkout v5 composite
actions/setup-node v6 composite

code-scan-action/action.yml actions

dist/index.js node20 javascript

examples/golang-provider/go.mod go

github.com/sashabaranov/go-openai v1.37.0

code-scan-action/package-lock.json npm

103 dependencies

examples/claude-agent-sdk/advanced/workspace/package.json npm

examples/claude-agent-sdk/working-dir/sample-project/package.json npm

express ^5.2.1

examples/http-provider-auth-signature-jks/package.json npm

express ^5.1.0
jks-js ^1.1.4
promptfoo ^0.119.12

examples/websockets-streaming/server/package.json npm

nodemon ^3.1.11 development
cors ^2.8.5
dotenv ^16.6.1
express ^5.2.1
openai ^4.104.0
ws ^8.18.3

examples/f-score/requirements.txt pypi

datasets ==4.2.0
pandas ==2.3.3

examples/google-adk-example/requirements.txt pypi

google-adk *
python-dotenv *

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Summary