Data

Tools

Indexes

Applications

Experiments

Summary

An open API service for producing an overview of a list of open source projects.

https://github.com/benchflow-ai/benchflow

benchmark evals

Score: 18.73772670644766

Last synced: about 23 hours ago
JSON representation

Repository metadata:

Research infra for creating RL environments, post-training, and evals.

Owner metadata:

GitHub Events

Total
Last Year

Committers metadata

Last synced: 2 days ago

Total Commits: 1,278
Total Committers: 28
Avg Commits per committer: 45.643
Development Distribution Score (DDS): 0.599

Commits in past year: 1,073
Committers in past year: 22
Avg Commits per committer in past year: 48.773
Development Distribution Score (DDS) in past year: 0.548

Name Email Commits
Xiangyi Li x****i@b****i 513
Bingran You b****u@b****u 243
kywch c****g@g****m 164
kirk k****k@b****m 158
Yimin 4****n 67
Yifeng He y****n@g****m 22
Test User t****t@e****m 21
devin-ai-integration[bot] 1****] 14
Bo Liu b****s@g****m 11
Zonglin DI e****1@g****m 10
kk 6****j 10
zjw49246 z****1@g****m 10
Zhijin Fang z****g@Z****l 5
symphony-bot s****a@g****m 5
Chujun Tao c****t@a****u 4
clawsbench c****h@l****l 4
Wenbo Chen w****n@g****u 3
Jason j****n@a****m 2
Shenghan Zheng z****1@g****m 2
Zhijin Fang d****g@g****m 2
Jiarui Liu 5****3 1
Nathan Habib 3****B 1
Noah Beito n****o@m****m 1
Tom Dörr t****6@g****m 1
VaisakhiMishra v****a@g****m 1
Yuchen Wu 7****u 1
explicitcontextualunderstanding e****g@g****m 1
kk-xuhj a****j@1****m 1

Issue and Pull Request metadata

Last synced: about 23 hours ago

Total issues: 69
Total pull requests: 236
Average time to close issues: 23 days
Average time to close pull requests: 2 days
Total issue authors: 36
Total pull request authors: 34
Average comments per issue: 1.46
Average comments per pull request: 2.21
Merged pull request: 160
Bot issues: 0
Bot pull requests: 0

Past year issues: 46
Past year pull requests: 206
Past year average time to close issues: 26 days
Past year average time to close pull requests: 1 day
Past year issue authors: 14
Past year pull request authors: 28
Past year average comments per issue: 1.43
Past year average comments per pull request: 2.51
Past year merged pull request: 136
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/benchflow-ai/benchflow

Top Issue Authors

  • Benjamin-eecs (13)
  • bingran-you (10)
  • kywch (5)
  • xdotli (4)
  • EYH0602 (3)
  • newlearnt (2)
  • yangziao56 (2)
  • Jason-Tree (2)
  • abderrahmane-br (1)
  • org-davidi (1)
  • shubham2345 (1)
  • kennethg3haynesq (1)
  • daniel7206-capponit (1)
  • BeniSisaki (1)
  • velezdouglasy-73 (1)

Top Pull Request Authors

  • bingran-you (81)
  • xdotli (57)
  • Yiminnn (24)
  • kk-xuhj (15)
  • kywch (13)
  • Wenbo11 (7)
  • shubham2345 (4)
  • tulerfeng (3)
  • Benjamin-eecs (3)
  • ljr145733 (2)
  • AdnanElAssadi (2)
  • Polarisyjr (2)
  • JeremyJC67 (2)
  • explicitcontextualunderstanding (1)
  • ElegantLin (1)

Top Issue Labels

  • P2 (11)
  • area:eval (8)
  • bug (8)
  • enhancement (7)
  • area:rollout (5)
  • reviewing (5)
  • status:in-progress (4)
  • reproduced (4)
  • P1 (4)
  • area:diagnostics (4)
  • question (3)
  • status:triage (1)
  • status:blocked (1)
  • area:dashboard (1)
  • area:sandbox (1)

Top Pull Request Labels

  • P1 (16)
  • review:pending (16)
  • enhancement (13)
  • status:ready (13)
  • area:eval (12)
  • bug (10)
  • status:blocked (8)
  • P2 (7)
  • area:rollout (6)
  • review:changes-requested (6)
  • area:diagnostics (6)
  • area:sandbox (3)
  • status:in-progress (2)

Package metadata

proxy.golang.org: github.com/benchflow-ai/benchflow

pypi.org: benchflow

Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.

  • Homepage: https://github.com/benchflow-ai/benchflow
  • Documentation: https://benchflow.readthedocs.io/
  • Licenses: Apache-2.0
  • Latest release: 0.7.11 (published 17 days ago)
  • Last Synced: 2026-10-31T15:35:33.376Z (2 days ago)
  • Versions: 155
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 9,845 Last month
  • Rankings:
    • Dependent packages count: 9.727%
    • Average: 32.241%
    • Dependent repos count: 54.756%
  • Maintainers (2)

Dependencies

.github/workflows/internal-preview-release.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • actions/download-artifact d3f86a106a0bac45b974a628896c90dbdf5c8093 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
.github/workflows/manifest-parity.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
benchmarks/clawsbench/tasks/archive-amazon-shipping/environment/claw-gmail/pyproject.toml pypi
  • click >=8.0
  • faker >=22.0
  • fastapi >=0.110.0
  • httpx >=0.27.0
  • jinja2 >=3.1
  • pydantic >=2.0
  • python-multipart >=0.0.6
  • sqlalchemy >=2.0
  • uvicorn [standard]>=0.27.0
.agents/skills/benchflow/tasks/create-simple-task/environment/Dockerfile docker
  • ubuntu 24.04 build
.agents/skills/benchflow/tasks/benchflow-knowledge/environment/Dockerfile docker
  • ubuntu 24.04 build
docs/examples/task-md/real-skillsbench/weighted-gdp-calc/environment/Dockerfile docker
  • ubuntu 24.04 build
tests/fixtures/skillsbench_slice/lake-warming-attribution/environment/Dockerfile docker
  • ubuntu 24.04 build
docs/examples/task-md/generated-skill-eval/models-as-skills/topo-sort-with-cycle-detection/environment/Dockerfile docker
  • python 3.12-slim build
docs/examples/task-md/real-skillsbench/3d-scan-calc/environment/Dockerfile docker
  • ubuntu 24.04 build
docs/examples/task-md/real-skillsbench/citation-check/environment/Dockerfile docker
  • ubuntu 24.04 build
src/benchflow/demo_task/environment/Dockerfile docker
  • ubuntu 24.04 build
tests/fixtures/skillsbench_slice/r2r-mpc-control/environment/Dockerfile docker
  • ubuntu 24.04 build
.github/workflows/integration-light.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • actions/setup-python a26af69be951a213d495a4c3e4e4022e16d87065 composite
  • actions/upload-artifact 65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
.github/workflows/integration-scope.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • actions/download-artifact d3f86a106a0bac45b974a628896c90dbdf5c8093 composite
  • actions/setup-python a26af69be951a213d495a4c3e4e4022e16d87065 composite
  • actions/upload-artifact 65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
docs/examples/nanofirm-task/environment/Dockerfile docker
  • python 3.12-slim build
services/trajectory_upload/Dockerfile docker
  • ghcr.io/astral-sh/uv python3.12-bookworm-slim@sha256 build
.devcontainer/Dockerfile docker
  • node 22 build
pyproject.toml pypi
  • agent-client-protocol >=0.10
  • anyio >=4.0
  • google-cloud-aiplatform >=1.133.0,<2.0
  • httpx >=0.27.0
  • litellm [proxy]==1.91.0
  • packaging >=24
  • pydantic >=2.7
  • pyyaml >=6.0
  • rich >=13.0
  • tomli-w >=1.0
  • typer >=0.9
tests/fixtures/skillsbench_slice/flood-risk-analysis/environment/Dockerfile docker
  • ubuntu 24.04 build
.github/workflows/public-release.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
.github/workflows/test.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
benchmarks/clawsbench/tasks/archive-amazon-shipping/environment/Dockerfile docker
  • kywch/smolclaws-base latest build
docs/examples/task-md/user-runtime/private-facts-nudges/environment/Dockerfile docker
  • python 3.12-slim build
tests/fixtures/digest/alpha-task/environment/Dockerfile docker
  • ubuntu 24.04 build
tests/conformance/acp_smoke/environment/Dockerfile docker
  • ubuntu 24.04 build
tests/fixtures/skillsbench_slice/syzkaller-ppdev-syzlang/environment/Dockerfile docker
  • ubuntu 24.04 build
docs/examples/task-md/generated-skill-eval/models-as-skills/regex-email-parser/environment/Dockerfile docker
  • python 3.12-slim build
uv.lock pypi
  • 238 dependencies
.github/workflows/deploy-trajectory-upload.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • azure/login a457da9ea143d694b1b9c7c869ebb04ebe844ef5 composite
.github/workflows/integration-final-review.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • actions/download-artifact d3f86a106a0bac45b974a628896c90dbdf5c8093 composite
  • actions/github-script 60a0d83039c74a4aee543508d2ffcb1c3799cdea composite
  • actions/setup-python a26af69be951a213d495a4c3e4e4022e16d87065 composite
  • actions/upload-artifact 65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 composite
  • astral-sh/setup-uv caf0cab7a618c569241d31dcd442f54681755d39 composite
docs/examples/task-md/generated-skill-eval/models-as-skills/optimize-quadratic-to-nlogn/environment/Dockerfile docker
  • python 3.12-slim build
tests/fixtures/skillsbench_slice/suricata-custom-exfil/environment/Dockerfile docker
  • jasonish/suricata 7.0.11 build
src/benchflow/sandbox/setup.py pypi
tests/examples/hello-world-task/environment/Dockerfile docker
  • ubuntu 24.04 build