Data

Tools

Indexes

Applications

Experiments

Summary

An open API service for producing an overview of a list of open source projects.

https://github.com/promptfoo/promptfoo

ci ci-cd cicd evaluation evaluation-framework llm llm-eval llm-evaluation llm-evaluation-framework llmops pentesting prompt-engineering prompt-testing prompts rag red-teaming testing vulnerability-scanners

Score: 30.614712037589577

Last synced: about 11 hours ago
JSON representation

Repository metadata:

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Owner metadata:

Committers metadata

Last synced: about 12 hours ago

Total Commits: 9,413
Total Committers: 324
Avg Commits per committer: 29.052
Development Distribution Score (DDS): 0.691

Commits in past year: 4,064
Committers in past year: 154
Avg Commits per committer in past year: 26.39
Development Distribution Score (DDS) in past year: 0.677

Name Email Commits
ianw_github i****b@i****m 2913
mldangelo 7****o 2704
renovate[bot] 2****] 1124
dependabot[bot] 4****] 465
sklein12 s****e@p****v 321
will-holley 8****y 221
MrFlounder k****s@g****m 190
faizanminhas 5****s 183
gru-agent[bot] 1****] 105
jameshiester 5****r 87
jbeckwith j****h@g****m 84
promptfoobot[bot] 2****] 48
kkahadze 8****e 46
danenania d****e@p****v 40
yash2998chhabria y****a@g****m 39
AI Simply Explained a****9@g****m 38
vedantr v****r 36
use-tusk[bot] 1****] 34
addelong a****n@p****v 31
Ryanba 9****h 30
vsauter 4****r 29
abrayne 4****e 20
minhle1291 m****1@g****m 19
Anubhav Anand 7****d 16
devin-ai-integration[bot] 1****] 15
Tabs t****b@l****m 14
Yufeng He 4****g 13
Evan Bonsignori e****i@g****m 11
Swarna Podila 1****p 11
NekoPunch 9****r 10
and 294 more...

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 805
Total pull requests: 4,605
Average time to close issues: 16 days
Average time to close pull requests: 2 days
Total issue authors: 428
Total pull request authors: 310
Average comments per issue: 1.61
Average comments per pull request: 1.22
Merged pull request: 3,168
Bot issues: 4
Bot pull requests: 899

Past year issues: 73
Past year pull requests: 920
Past year average time to close issues: 6 days
Past year average time to close pull requests: 3 days
Past year issue authors: 63
Past year pull request authors: 130
Past year average comments per issue: 0.71
Past year average comments per pull request: 2.27
Past year merged pull request: 418
Past year bot issues: 1
Past year bot pull requests: 200

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/promptfoo/promptfoo

Top Issue Authors

  • jamesbraza (18)
  • albertlieyingadrian (17)
  • aantn (15)
  • mldangelo (13)
  • typpo (12)
  • pelikhan (10)
  • efung (10)
  • SysOverdrive (10)
  • romaintoub (8)
  • sbichenko (8)
  • rvernica (8)
  • zhlmmc (8)
  • chrismaltais (8)
  • sangwoo-joh (7)
  • AmineDjeghri (7)

Top Pull Request Authors

  • mldangelo (1,252)
  • typpo (1,041)
  • gru-agent[bot] (391)
  • dependabot[bot] (310)
  • sklein12 (251)
  • will-holley (143)
  • renovate[bot] (142)
  • MrFlounder (114)
  • mldangelo-oai (95)
  • faizanminhas (86)
  • AISimplyExplained (40)
  • vedantr (37)
  • devin-ai-integration[bot] (28)
  • ianw-oai (28)
  • jameshiester-oai (27)

Top Issue Labels

  • bug (33)
  • enhancement (30)
  • question (26)
  • good first issue (6)
  • dependencies (5)
  • in-progress (5)
  • Open Source (4)
  • javascript (4)
  • documentation (3)
  • help wanted (2)
  • wontfix (2)
  • regression (1)
  • good-first-issue (1)
  • codex (1)
  • Feature Request (1)

Top Pull Request Labels

  • dependencies (310)
  • javascript (284)
  • codex (100)
  • codex-automation (42)
  • python (19)
  • autorelease: pending (6)
  • bug (6)
  • documentation (5)
  • github_actions (5)
  • enhancement (4)
  • sourcery (3)
  • autorelease: tagged (2)
  • in-progress (1)
  • good first issue (1)
  • question (1)

Package metadata

proxy.golang.org: github.com/Promptfoo/promptfoo

  • Homepage: https://github.com/Promptfoo/promptfoo
  • Documentation: https://pkg.go.dev/github.com/Promptfoo/promptfoo#section-documentation
  • Licenses: mit
  • Latest release: (published 1 day ago)
  • Last Synced: 2026-08-25T13:11:14.254Z (1 day ago)
  • Versions: 0
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Rankings:
    • Dependent packages count: 4.52%
    • Average: 4.672%
    • Dependent repos count: 4.824%
proxy.golang.org: github.com/promptfoo/promptfoo/examples/golang-provider

Package main implements a promptfoo provider that uses OpenAI's API.

proxy.golang.org: github.com/promptfoo/promptfoo

  • Homepage: https://github.com/promptfoo/promptfoo
  • Documentation: https://pkg.go.dev/github.com/promptfoo/promptfoo#section-documentation
  • Licenses: mit
  • Latest release: v0.103.14 (published over 1 year ago)
  • Last Synced: 2026-08-25T14:07:15.669Z (1 day ago)
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Rankings:
    • Dependent packages count: 6.194%
    • Average: 6.402%
    • Dependent repos count: 6.609%
npmjs.org: promptfoo

LLM eval & testing toolkit

  • Homepage: https://promptfoo.dev
  • Licenses: MIT
  • Latest release: 0.122.0 (published 22 days ago)
  • Last Synced: 2026-08-25T14:06:18.733Z (1 day ago)
  • Versions: 419
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 2,435,183 Last month
  • Rankings:
    • Downloads: 0.804%
    • Stargazers count: 2.307%
    • Forks count: 4.269%
    • Dependent repos count: 10.302%
    • Average: 13.908%
    • Dependent packages count: 51.857%
  • Maintainers (5)
npmjs.org: @alexcarol/promptfoo

LLM eval & testing toolkit

  • Homepage: https://github.com/promptfoo/promptfoo#readme
  • Status: removed
  • Licenses: MIT
  • Latest release: 0.119.11 (published 9 months ago)
  • Last Synced: 2026-08-25T14:06:18.119Z (1 day ago)
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 9 Last month
  • Rankings:
    • Dependent repos count: 23.384%
    • Average: 28.555%
    • Dependent packages count: 33.727%
  • Maintainers (1)

Dependencies

.github/workflows/main.yml actions
  • actions/checkout v3 composite
  • actions/setup-node v3 composite
  • bahmutov/npm-install v1 composite
package-lock.json npm
  • 495 dependencies
package.json npm
  • @types/async ^3.2.20 development
  • @types/cache-manager ^4.0.2 development
  • @types/cache-manager-fs-hash ^0.0.1 development
  • @types/cli-progress ^3.11.0 development
  • @types/cors ^2.8.13 development
  • @types/debounce ^1.2.1 development
  • @types/express ^4.17.17 development
  • @types/glob ^8.1.0 development
  • @types/jest ^29.5.1 development
  • @types/js-yaml ^4.0.5 development
  • @types/node-fetch ^2.6.4 development
  • @types/nunjucks ^3.2.2 development
  • @types/opener ^1.4.0 development
  • @types/semver ^7.5.0 development
  • babel-jest ^29.5.0 development
  • jest ^29.5.0 development
  • jest-watch-typeahead ^2.2.2 development
  • prettier ^2.8.8 development
  • ts-jest ^29.1.0 development
  • ts-node ^10.9.1 development
  • typescript ^5.0.4 development
  • @anthropic-ai/sdk ^0.5.2
  • @apidevtools/json-schema-ref-parser ^10.1.0
  • async ^3.2.4
  • cache-manager ^4.1.0
  • cache-manager-fs-hash ^1.0.0
  • chalk ^4.1.2
  • cli-progress ^3.12.0
  • cli-table3 ^0.6.3
  • commander ^10.0.1
  • cors ^2.8.5
  • csv-parse ^5.3.8
  • csv-stringify ^6.3.2
  • debounce ^1.2.1
  • express ^4.18.2
  • glob ^10.2.6
  • js-yaml ^4.1.0
  • node-fetch ^2.6.7
  • nunjucks ^3.2.4
  • opener ^1.5.2
  • replicate ^0.12.3
  • rouge ^1.0.3
  • semver ^7.5.3
  • socket.io ^4.6.1
  • tiny-invariant ^1.3.1
  • winston ^3.8.2
Dockerfile docker
  • node 16-alpine build
examples/openai-agents-basic/package.json npm
  • @openai/agents ^0.3.3
  • zod ^3.25.76
examples/redteam-langchain/requirements.txt pypi
  • PyYAML ==6.0.3
  • SQLAlchemy ==2.0.43
  • aiohappyeyeballs ==2.6.1
  • aiohttp ==3.13.0
  • aiosignal ==1.4.0
  • annotated-types ==0.7.0
  • anyio ==4.11.0
  • attrs ==25.4.0
  • certifi ==2025.10.5
  • charset-normalizer ==3.4.3
  • dataclasses-json ==0.6.7
  • distro ==1.9.0
  • frozenlist ==1.8.0
  • h11 ==0.16.0
  • httpcore ==1.0.9
  • httpx ==0.28.1
  • httpx-sse ==0.4.2
  • idna ==3.10
  • jiter ==0.11.0
  • jsonpatch ==1.33
  • jsonpointer ==3.0.0
  • langchain ==0.3.27
  • langchain-community ==0.3.31
  • langchain-core ==0.3.80
  • langchain-openai ==0.3.35
  • langchain-text-splitters ==0.3.11
  • langsmith ==0.4.33
  • marshmallow ==3.26.1
  • multidict ==6.7.0
  • mypy_extensions ==1.1.0
  • numpy ==2.3.3
  • openai ==2.2.0
  • orjson ==3.11.3
  • packaging ==25.0
  • propcache ==0.4.0
  • pydantic ==2.12.0
  • pydantic-settings ==2.11.0
  • pydantic_core ==2.41.1
  • python-dotenv ==1.1.1
  • regex ==2025.9.18
  • requests ==2.32.5
  • requests-toolbelt ==1.0.0
  • sniffio ==1.3.1
  • tenacity ==9.1.2
  • tiktoken ==0.12.0
  • tqdm ==4.67.1
  • typing-inspect ==0.9.0
  • typing-inspection ==0.4.2
  • typing_extensions ==4.15.0
  • urllib3 ==2.5.0
  • yarl ==1.22.0
  • zstandard ==0.25.0
code-scan-action/package.json npm
  • @types/jest ^30.0.0 development
  • @types/node ^24.10.1 development
  • @vercel/ncc ^0.38.4 development
  • typescript ^5.9.3 development
  • @actions/core ^1.11.1
  • @actions/exec ^1.1.1
  • @actions/github ^6.0.1
  • @octokit/auth-app ^8.1.2
  • @octokit/rest ^22.0.1
.github/workflows/docker.yml actions
  • actions/checkout v5 composite
  • actions/download-artifact v6 composite
  • actions/setup-node v6 composite
  • actions/upload-artifact v5 composite
  • docker/build-push-action v6 composite
  • docker/login-action v3 composite
  • docker/metadata-action v5 composite
  • docker/setup-buildx-action v3 composite
  • docker/setup-qemu-action v3 composite
.github/workflows/image-actions.yml actions
  • actions/checkout v5 composite
  • calibreapp/image-actions main composite
examples/redteam-tracing-example/package.json npm
  • express ^5.2.1
examples/openai-agents/requirements.txt pypi
  • openai-agents ==0.1.0
  • pydantic ==2.12.0
.github/workflows/deploy-launcher.yml actions
  • actions/checkout v5 composite
  • actions/setup-node v6 composite
  • cloudflare/wrangler-action v3 composite
examples/redteam-medical-agent/package.json npm
  • cors ^2.8.5
  • dotenv ^17.2.3
  • express ^5.1.0
  • openai ^5.23.2
examples/simple-mcp/package.json npm
  • promptfoo * development
  • @modelcontextprotocol/sdk ^1.22.0
  • zod ^3.25.76
.github/workflows/tusk-test-runner-vitest-unit-tests.yml actions
  • Use-Tusk/test-runner v1 composite
  • actions/checkout v5 composite
  • actions/setup-node v6 composite
.github/workflows/validate-pr-title.yml actions
  • amannn/action-semantic-pull-request v6 composite
.github/workflows/promptfoo-code-scan.yml actions
  • actions/checkout v4 composite
  • promptfoo/code-scan-action v0 composite
.devcontainer/docker-compose.yml docker
site/package.json npm
  • @calcom/embed-react ^1.5.3 development
  • @docusaurus/core ^3.9.2 development
  • @docusaurus/module-type-aliases ^3.9.2 development
  • @docusaurus/plugin-client-redirects ^3.9.2 development
  • @docusaurus/plugin-content-docs ^3.9.2 development
  • @docusaurus/preset-classic ^3.9.2 development
  • @docusaurus/theme-mermaid ^3.9.2 development
  • @mdx-js/react ^3.1.1 development
  • @monaco-editor/react ^4.7.0 development
  • @mui/icons-material ^7.3.5 development
  • @segment/ajv-human-errors ^2.15.0 development
  • @swc/core ^1.15.3 development
  • @swc/jest ^0.2.39 development
  • @tsconfig/docusaurus ^2.0.7 development
  • canvas-confetti ^1.9.4 development
  • clsx ^2.1.1 development
  • dedent ^1.7.0 development
  • docusaurus-plugin-image-zoom ^3.0.1 development
  • framer-motion ^12.23.24 development
  • gray-matter ^4.0.3 development
  • lucide-react ^0.555.0 development
  • prism-react-renderer ^2.4.1 development
  • react 19.2.0 development
  • react-dom 19.2.0 development
  • satori ^0.18.3 development
  • sharp ^0.34.5 development
  • typescript ^5.9.3 development
  • @mui/material ^7.3.5
  • @mui/system ^7.3.5
  • @types/react 19.2.7
  • @types/react-dom 19.2.3
  • react-countup ^6.5.3
  • swiper ^12.0.0
examples/redteam-chatbot/package.json npm
  • express ^5.1.0
  • promptfoo ^0.119.12
examples/redteam-mcp-agent/package.json npm
  • @faker-js/faker ^9.9.0
  • @modelcontextprotocol/sdk ^1.22.0
  • dotenv ^17.2.3
  • openai ^5.23.2
.github/workflows/release-please.yml actions
  • actions/checkout v5 composite
  • actions/checkout v4 composite
  • actions/setup-node v6 composite
  • googleapis/release-please-action v4 composite
.github/workflows/tusk-test-runner-app-vitest-unit-tests.yml actions
  • Use-Tusk/test-runner v1 composite
  • actions/checkout v5 composite
  • actions/setup-node v6 composite
code-scan-action/action.yml actions
  • dist/index.js node20 javascript
code-scan-action/package-lock.json npm
  • 103 dependencies
examples/claude-agent-sdk/advanced/workspace/package.json npm
examples/claude-agent-sdk/working-dir/sample-project/package.json npm
  • express ^5.2.1
.github/workflows/release-please-sha-drift.yml actions
  • actions/checkout df4cb1c069e1874edd31b4311f1884172cec0e10 composite
examples/config-node-module-package/node_modules/testpackage/package.json npm
examples/openai-agents-advanced/package.json npm
  • @openai/agents ^0.11.3
  • zod ^4.3.6
examples/eval-image-classification/requirements.txt pypi
  • numpy ==1.26.4
  • pandas ==2.3.3
  • pillow ==12.3.0
  • tensorflow ==2.20.0
examples/integration-google-adk/requirements.txt pypi
  • google-adk >=1.32.0,<2
  • opentelemetry-api >=1.30.0
  • opentelemetry-exporter-otlp-proto-http >=1.30.0
  • opentelemetry-sdk >=1.30.0
examples/provider-golang/go.mod go
  • github.com/sashabaranov/go-openai v1.37.0
examples/config-stateful-session-management/package.json npm
  • express ^5.2.1
  • openai ^7.1.0
  • uuid ^14.0.0
examples/integration-vercel/ai-sdk/package.json npm
  • @ai-sdk/openai ^3.0.41
  • ai ^6.0.190
examples/integration-strands-agents/requirements.txt pypi
  • pydantic >=2.0.0
  • strands-agents >=1.0.0,<2.0.0
examples/provider-python/requirements.txt pypi
  • annotated-types ==0.6.0
  • anyio ==4.2.0
  • certifi ==2025.10.5
  • distro ==1.9.0
  • exceptiongroup ==1.2.0
  • h11 ==0.16.0
  • httpcore ==1.0.2
  • httpx ==0.26.0
  • idna ==3.15
  • openai ==2.3.0
examples/integration-opentelemetry/python/requirements.txt pypi
  • opentelemetry-api >=1.28.0
  • opentelemetry-exporter-otlp-proto-http >=1.28.0
  • opentelemetry-sdk >=1.28.0
  • opentelemetry-semantic-conventions >=0.49b0
examples/integration-pydantic-ai/requirements.txt pypi
  • openai ==2.38.0
  • pydantic ==2.13.4
  • pydantic-ai ==1.106.0
  • python-dotenv ==1.2.2
examples/config-websockets/basic/test-server/package.json npm
  • ws ^8.21.1
examples/config-websockets/streaming/server/package.json npm
  • nodemon ^3.1.11 development
  • cors ^2.8.6
  • dotenv ^17.2.3
  • express ^5.2.1
  • openai ^7.1.0
  • ws ^8.21.1
examples/config-ts/package.json npm
  • @types/node ^25.1.0 development
  • promptfoo latest development
  • openai ^7.1.0
  • tsx ^4.21.0
  • zod ^4.3.6
pnpm-workspace.yaml npm
examples/eval-bert-score/requirements.txt pypi
  • bert-score >=0.3.13
  • torch >=1.0.0
  • transformers >=4.0.0
.github/workflows/release-please-format.yml actions
  • actions/checkout df4cb1c069e1874edd31b4311f1884172cec0e10 composite
  • actions/create-github-app-token bcd2ba49218906704ab6c1aa796996da409d3eb1 composite
  • actions/setup-node 48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e composite
examples/provider-http/auth-signature/package.json npm
  • express ^5.2.1
  • express-rate-limit ^8.2.1
  • promptfoo ^0.121.0
examples/redteam-indirect-web-pwn/package.json npm
  • openai ^7.1.0
examples/eval-f-score/requirements.txt pypi
  • datasets ==4.2.0
  • pandas ==2.3.3
examples/eval-rag-full/requirements.txt pypi
  • chromadb ==1.5.9
  • langchain-chroma ==1.1.0
  • langchain-community ==0.4.1
  • langchain-core ==1.3.3
  • langchain-openai ==1.2.1
  • langchain-text-splitters ==1.1.2
  • pypdf ==6.15.0
  • tqdm ==4.67.3
scripts/modelaudit_schema_requirements.txt pypi
  • annotated-types ==0.7.0
  • modelaudit ==0.2.52
  • pydantic ==2.13.4
  • pydantic-core ==2.46.4
  • typing-extensions ==4.15.0
  • typing-inspection ==0.4.2
examples/integration-opentelemetry/javascript/package.json npm
  • @opentelemetry/api ^1.9.0
  • @opentelemetry/exporter-trace-otlp-http ^0.221.0
  • @opentelemetry/resources ^2.5.0
  • @opentelemetry/sdk-trace-node ^2.5.0
  • @opentelemetry/semantic-conventions ^1.39.0
examples/redteam-api-top-10/pyproject.toml pypi
  • anthropic >=0.40.0
  • fastapi >=0.115.0
  • httpx >=0.27.0
  • mcp >=1.0.0
  • pyjwt >=2.9.0
  • python-dotenv >=1.0.0
  • uvicorn [standard]>=0.32.0
.github/workflows/validate-renovate-config.yml actions
  • actions/checkout df4cb1c069e1874edd31b4311f1884172cec0e10 composite
examples/provider-golang/go.sum go
  • github.com/sashabaranov/go-openai v1.37.0
examples/integration-browser/headless/requirements.txt pypi
  • gradio >=5.49.1
  • playwright >=1.55.0
examples/integration-crewai/requirements.txt pypi
  • crewai >=0.203.0
examples/integration-docker/code-generation-sandbox/requirements.txt pypi
  • certifi ==2025.10.5
  • charset-normalizer ==3.3.2
  • docker ==7.1.0
  • epicbox ==1.1.0
  • idna ==3.15
  • python-dateutil ==2.9.0.post0
  • requests ==2.33.0
  • six ==1.16.0
  • structlog ==24.2.0
  • urllib3 ==2.7.0
examples/integration-jest/package.json npm
  • @types/jest ^30.0.0 development
  • jest ^30.2.0 development
  • ts-jest ^29.4.6 development
  • typescript ^5.9.3 development
  • vitest ^4.0.18 development
examples/integration-langchain/requirements.txt pypi
  • langchain-core ==1.4.2
  • langchain-openai ==1.1.14
  • openai ==2.26.0
examples/provider-http/auth-signature-jks/package.json npm
  • express ^5.2.1
  • express-rate-limit ^8.2.1
  • jks-js ^1.1.5
  • promptfoo ^0.121.0
examples/provider-http/auth-signature-pfx/package.json npm
  • express ^5.2.1
  • express-rate-limit ^8.2.1
  • pem ^1.15.1
  • promptfoo ^0.121.0
examples/config-node-package/package.json npm
  • promptfoo latest
examples/config-node-package-typescript/package.json npm
  • @types/node ^24.10.9 development
  • tsx ^4.21.0 development
  • typescript ^7.0.0 development
  • promptfoo latest