https://github.com/confident-ai/deepeval
evaluation-framework evaluation-metrics llm-evaluation llm-evaluation-framework llm-evaluation-metrics python
Score: 30.538927372657035
Last synced: about 6 hours ago
JSON representation
Repository metadata:
The LLM Evaluation Framework
- Host: GitHub
- URL: https://github.com/confident-ai/deepeval
- Owner: confident-ai
- License: apache-2.0
- Created: 2023-08-10T05:35:04.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2026-10-27T16:30:35.000Z (3 days ago)
- Last Synced: 2026-10-28T18:57:06.785Z (2 days ago)
- Topics: evaluation-framework, evaluation-metrics, llm-evaluation, llm-evaluation-framework, llm-evaluation-metrics, python
- Language: Python
- Homepage: https://deepeval.com
- Size: 205 MB
- Stars: 19,169
- Watchers: 65
- Forks: 2,145
- Open Issues: 788
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md
- Citation: CITATION.cff
- Maintainers: MAINTAINERS.md
Owner metadata:
- Name: Confident AI
- Login: confident-ai
- Email:
- Kind: organization
- Description:
- Website: www.confident-ai.com
- Location:
- Twitter:
- Company:
- Icon url: https://avatars.githubusercontent.com/u/130858411?v=4
- Repositories: 13
- Last Synced at: 2026-10-22T12:42:30.365Z
- Profile URL: https://github.com/confident-ai
GitHub Events
Total
- Commit comment event: 3
- Create event: 244
- Delete event: 173
- Discussion event: 4
- Fork event: 677
- Issue comment event: 1624
- Issues event: 398
- Member event: 3
- Pull request event: 1257
- Pull request review comment event: 77
- Pull request review event: 53
- Push event: 1859
- Release event: 15
- Watch event: 6983
- Total: 13370
Last Year
- Create event: 54
- Delete event: 39
- Discussion event: 4
- Fork event: 154
- Issue comment event: 373
- Issues event: 116
- Member event: 1
- Pull request event: 259
- Pull request review comment event: 39
- Pull request review event: 25
- Push event: 487
- Watch event: 1513
- Total: 3064
Committers metadata
Last synced: 1 day ago
Total Commits: 8,208
Total Committers: 339
Avg Commits per committer: 24.212
Development Distribution Score (DDS): 0.622
Commits in past year: 2,117
Committers in past year: 109
Avg Commits per committer in past year: 19.422
Development Distribution Score (DDS) in past year: 0.596
| Name | Commits | |
|---|---|---|
| Jeffrey Ip | j****p@c****m | 3101 |
| A-Vamshi | v****y@i****m | 1257 |
| Jacky Wong | c****g@g****m | 858 |
| Kritin_Vongthongsri | k****v@p****u | 788 |
| Mayank Solanki | b****u@g****m | 731 |
| BloggerBust | t****n@b****a | 388 |
| tanayvaswani | v****9@g****m | 168 |
| trevor-cai | t****r@c****m | 65 |
| Anindyadeep | p****p@g****m | 47 |
| Anindyadeep | a****a@p****n | 35 |
| Vasilije | 8****0 | 26 |
| Pratyush-exe | p****5@g****m | 23 |
| Sidhaarth Sredharan | s****7@g****m | 19 |
| john | j****n@s****m | 18 |
| Aman Gokrani | a****i@g****m | 17 |
| Sai-Suraj-27 | s****9@g****m | 15 |
| fetz236 | 5****6 | 14 |
| Peilun Li | p****l@z****m | 13 |
| tony-confidentai | 3****i | 12 |
| Jack Luar | 3****s | 11 |
| ykocaogullar | y****r@g****m | 10 |
| Vamshi Adimalla | v****a@V****l | 9 |
| Vytenis Šliogeris | v****s@n****m | 9 |
| Frederico Schuh | f****h@g****m | 9 |
| Serghei Iakovlev | g****t@s****l | 7 |
| Jon Bennion | 1****b | 7 |
| Jan F. | 5****4 | 7 |
| Andrea Romano | 1****o | 7 |
| Alex Maggioni | 9****i | 6 |
| Christian Bernhard | 4****d | 6 |
| and 309 more... | ||
Issue and Pull Request metadata
Last synced: 1 day ago
Total issues: 559
Total pull requests: 2,283
Average time to close issues: 21 days
Average time to close pull requests: 5 days
Total issue authors: 394
Total pull request authors: 475
Average comments per issue: 1.14
Average comments per pull request: 1.38
Merged pull request: 1,466
Bot issues: 1
Bot pull requests: 2
Past year issues: 136
Past year pull requests: 513
Past year average time to close issues: 9 days
Past year average time to close pull requests: 14 days
Past year issue authors: 86
Past year pull request authors: 236
Past year average comments per issue: 1.27
Past year average comments per pull request: 1.77
Past year merged pull request: 84
Past year bot issues: 1
Past year bot pull requests: 0
Top Issue Authors
- Asthenia0412 (24)
- ColabDog (24)
- penguine-ip (19)
- prescod (11)
- sahniaditya007 (6)
- piseabhijeet (5)
- AndresPrez (5)
- luarss (4)
- tjasmin111 (4)
- feiiiiii5 (4)
- CAW-nz (4)
- Sara-Hossny (4)
- kittimzhe (4)
- jmaczan (3)
- BlueX888 (3)
Top Pull Request Authors
- penguine-ip (566)
- kritinv (430)
- A-Vamshi (105)
- spike-spiegel-21 (101)
- ColabDog (62)
- Asthenia0412 (38)
- BloggerBust (29)
- john-lemmon-lime (17)
- Wilson0511 (15)
- luarss (15)
- feiiiiii5 (15)
- tanayvaswani (14)
- Sai-Suraj-27 (12)
- ramipellumbi (12)
- ChristianBernhard (12)
Top Issue Labels
- help wanted (17)
- enhancement (11)
- good first issue (5)
- documentation (3)
- bug (2)
- awaiting clarification (2)
- correctness (1)
- kind/enhancement (1)
- Flaky Tests (1)
- awaiting code fix (1)
- area/docs (1)
Top Pull Request Labels
- awaiting docs fix (4)
- awaiting code fix (3)
- typescript (2)
- Stale (1)
- v8.18.0 (1)
- java (1)
- awaiting clarification (1)
- awaiting fix (1)
- dependencies (1)
- >non-issue (1)
- Team:StorageEngine (1)
- backport (1)
- :StorageEngine/Logs (1)
- auto-merge-without-approval (1)
- new talk (1)
Package metadata
- Total packages: 5
-
Total downloads:
- conda: 357 total
- pypi: 2,584,198 last-month
- npm: 123,085 last-month
- Total dependent packages: 7 (may contain duplicates)
- Total dependent repositories: 1 (may contain duplicates)
- Total versions: 883
- Total maintainers: 3
proxy.golang.org: github.com/confident-ai/deepeval
- Homepage:
- Documentation: https://pkg.go.dev/github.com/confident-ai/deepeval#section-documentation
- Licenses: apache-2.0
- Latest release: v4.1.8+incompatible (published 3 months ago)
- Last Synced: 2026-10-29T08:32:33.360Z (1 day ago)
- Versions: 302
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Dependent packages count: 6.528%
- Average: 6.747%
- Dependent repos count: 6.966%
pypi.org: deepeval
The LLM Evaluation Framework
- Homepage:
- Documentation: https://deepeval.com
- Licenses: Apache-2.0
- Latest release: 4.3.2 (published 3 days ago)
- Last Synced: 2026-10-29T08:32:24.575Z (1 day ago)
- Versions: 531
- Dependent Packages: 7
- Dependent Repositories: 1
- Downloads: 2,584,198 Last month
-
Rankings:
- Stargazers count: 2.812%
- Downloads: 3.967%
- Forks count: 8.248%
- Average: 9.348%
- Dependent packages count: 10.095%
- Dependent repos count: 21.615%
- Maintainers (2)
npmjs.org: deepeval-ts
Deprecated — renamed to 'deepeval'
- Homepage: https://deepeval.com
- Status: deprecated
- Licenses: Apache-2.0
- Latest release: 1.0.0 (published 2 months ago)
- Last Synced: 2026-10-29T08:47:45.760Z (1 day ago)
- Versions: 21
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 3,469 Last month
-
Rankings:
- Downloads: 6.97%
- Average: 22.321%
- Dependent repos count: 24.554%
- Dependent packages count: 35.44%
- Maintainers (2)
npmjs.org: deepeval
The LLM Evaluation Framework for TypeScript
- Homepage: https://github.com/confident-ai/deepeval#readme
- Licenses: apache-2.0
- Latest release: 0.9.24 (published 3 days ago)
- Last Synced: 2026-10-29T08:48:56.497Z (1 day ago)
- Versions: 26
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 119,616 Last month
-
Rankings:
- Dependent repos count: 22.7%
- Average: 27.71%
- Dependent packages count: 32.72%
- Maintainers (2)
anaconda.org: deepeval
DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which uses LLM-as-a-judge and other NLP models that run locally on your machine.
- Homepage: https://github.com/confident-ai/deepeval
- Licenses: Apache-2.0
- Latest release: 4.1.4 (published 3 months ago)
- Last Synced: 2026-07-31T13:03:38.958Z (3 months ago)
- Versions: 3
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 357 Total
-
Rankings:
- Dependent packages count: 35.874%
- Dependent repos count: 39.116%
- Average: 48.659%
- Downloads: 70.985%
Dependencies
.github/workflows/black.yml
actions
- actions/checkout v3 composite
- psf/black stable composite
docs/package.json
npm
- @docusaurus/module-type-aliases 2.4.1 development
- @docusaurus/core 2.4.1
- @docusaurus/preset-classic 2.4.1
- @mdx-js/react ^1.6.22
- clsx ^1.2.1
- prism-react-renderer ^1.3.5
- react ^17.0.2
- react-dom ^17.0.2
poetry.lock
pypi
- 617 dependencies
typescript/package-lock.json
npm
- 986 dependencies
.github/workflows/agent_skills_check.yml
actions
- actions/checkout v4 composite
- actions/setup-node v4 composite
.github/workflows/test_templates.yml
actions
- actions/checkout v3 composite
- actions/setup-python v4 composite
typescript/package.json
npm
- @ai-sdk/openai ^3.0.30 development
- @babel/core ^7.29.7 development
- @babel/preset-env ^7.29.7 development
- @babel/preset-typescript ^7.29.7 development
- @eslint/js ^9.38.0 development
- @google/genai ^2.8.0 development
- @jest/globals ^29.5.0 development
- @langchain/core 1.1.31 development
- @langchain/openai ^1.2.9 development
- @mastra/core ^1.51.0 development
- @mastra/observability ^1.16.1 development
- @openai/agents ^0.5.4 development
- @opentelemetry/core ^2.6.0 development
- @types/cli-progress ^3.11.0 development
- @types/jest ^29.5.0 development
- @types/node ^20.17.30 development
- @types/nunjucks ^3.2.6 development
- @types/papaparse ^5.3.16 development
- @types/react ^19.2.18 development
- @types/uuid ^9.0.0 development
- ai ^6.0.97 development
- babel-jest ^29.7.0 development
- eslint ^9.38.0 development
- eslint-config-prettier ^10.1.8 development
- globals ^16.4.0 development
- husky ^9.1.7 development
- jest ^29.5.0 development
- jiti ^2.6.1 development
- lint-staged ^16.2.6 development
- prettier 3.6.2 development
- ts-jest ^29.1.0 development
- ts-node ^10.9.2 development
- tsc-alias ^1.9.1 development
- typescript ^5.8.3 development
- typescript-eslint ^8.46.2 development
- vitest ^4.1.10 development
- @opentelemetry/api ^1.9.0
- @opentelemetry/exporter-trace-otlp-grpc ^0.207.0
- @opentelemetry/exporter-trace-otlp-proto ^0.211.0
- @opentelemetry/resources ^2.2.0
- @opentelemetry/sdk-trace-base ^2.2.0
- @opentelemetry/sdk-trace-node ^2.2.0
- @opentelemetry/semantic-conventions ^1.37.0
- axios ^1.6.0
- cli-progress ^3.12.0
- commander ^14.0.3
- csv-parser ^3.2.0
- dotenv ^16.5.0
- nunjucks ^3.2.4
- open ^9.1.0
- openai ^6.7.0
- ora ^6.3.0
- papaparse ^5.5.3
- posthog-node ^5.10.3
- uuid ^9.0.0
- zod ^4.3.6
docs/content/docs/(voice)/meta.json
cpan
docs/yarn.lock
npm
- 524 dependencies
docs/content/docs/conversation-simulator-voice-mode/meta.json
cpan
pyproject.toml
pypi
- anthropic * develop
- black ^25.1.0 develop
- chromadb * develop
- docx2txt * develop
- google-genai ^1.9.0 develop
- langchain * develop
- langchain-community * develop
- langchain-core * develop
- langchain-text-splitters * develop
- ollama * develop
- pandas * develop
- pre-commit ^4.3.0 develop
- pypdf * develop
- pysqlite3-binary ^0.5.4 develop
- ruff ^0.13.0 develop
- tiktoken * develop
- twine 5.1.1 develop
- crewai * integrations
- google-adk * integrations
- llama-index ^0.14.4 integrations
- openai-agents ^0.3.3 integrations
- openinference-instrumentation-google-adk * integrations
- pydantic-ai-slim * integrations
- langchain 1.2.4 langchain
- langchain-openai 1.1.7 langchain
- langgraph 1.0.7 langchain
- aiohttp *
- click >=8.0.0,<8.4.0
- grpcio ^1.67.1
- jinja2 *
- nest-asyncio *
- openai *
- opentelemetry-api ^1.24.0
- opentelemetry-sdk ^1.24.0
- portalocker *
- posthog >=5.4.0, <7.0.0
- posthog >=7.0.0, <8.0.0
- pydantic ^2.11.7
- pydantic-settings ^2.10.1
- pyfiglet *
- pyperclip ^1.8
- pytest *
- pytest-asyncio *
- pytest-repeat *
- pytest-rerunfailures *
- pytest-xdist *
- python >=3.9, <4.0
- python-dotenv ^1.1.1
- questionary >=2.0.0,<3.0.0
- requests ^2.31.0
- rich >=13.6.0, <15.0.0
- setuptools *
- tabulate ^0.9.0
- tenacity >=8.0.0,<=10.0.0
- textual >=0.80,<2.0
- tqdm ^4.66.1
- typer >=0.9,<1.0.0
- wheel *
.github/workflows/typescript_lint.yml
actions
- actions/checkout v3 composite
docs/content/docs/(agentic)/meta.json
cpan
.github/workflows/test_metrics.yml
actions
- actions/cache v3 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- snok/install-poetry v1 composite
.github/workflows/test_model_registry.yml
actions
- actions/checkout v3 composite
- actions/setup-python v4 composite
.github/workflows/typescript_confident.yml
actions
- actions/checkout v3 composite
.github/workflows/changelog.yml
actions
- actions/checkout v4 composite
- actions/setup-python v5 composite
- peter-evans/create-pull-request v6 composite
.github/workflows/pr-title-check.yml
actions
- amannn/action-semantic-pull-request v5 composite
.github/workflows/full_test_core_for_pr.yml
actions
- actions/cache v3 composite
- actions/checkout v4 composite
- actions/setup-python v4 composite
- snok/install-poetry v1 composite
.github/workflows/test_core.yml
actions
- actions/cache v3 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- snok/install-poetry v1 composite
.github/workflows/typescript_test.yml
actions
- actions/checkout v3 composite
.github/workflows/typescript_test_run_e2e.yml
actions
- actions/checkout v3 composite
.github/workflows/release.yml
actions
- actions/checkout v4 composite
- actions/setup-python v5 composite
docs/content/docs/(community)/meta.json
cpan
.github/workflows/typescript_integration.yml
actions
- actions/checkout v3 composite
docs/content/docs/(use-cases)/meta.json
cpan
docs/content/docs/golden-synthesizer/meta.json
cpan
.github/workflows/test_integrations.yml
actions
- actions/checkout v4 composite
- actions/setup-python v5 composite
- snok/install-poetry v1 composite
docs/content/docs/(custom)/meta.json
cpan
docs/content/integrations/models/meta.json
cpan
docs/content/integrations/others/meta.json
cpan
docs/content/tutorials/meta.json
cpan
.github/workflows/test_confident.yml
actions
- actions/cache v3 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- snok/install-poetry v1 composite
docs/content/docs/conversation-simulator/meta.json
cpan
docs/content/integrations/meta.json
cpan
docs/content/docs/evaluation-end-to-end-llm-evals/meta.json
cpan
docs/content/changelog/meta.json
cpan
docs/content/docs/(concepts)/(test-cases)/meta.json
cpan
docs/content/docs/(safety)/meta.json
cpan
docs/content/guides/meta.json
cpan
docs/content/docs/(mcp)/meta.json
cpan
docs/content/docs/(non-llm)/meta.json
cpan
docs/content/blog/meta.json
cpan
docs/content/docs/(algorithms)/meta.json
cpan
docs/content/docs/(generate-goldens)/meta.json
cpan
docs/content/docs/(metrics-others)/meta.json
cpan
docs/content/docs/(benchmarks)/meta.json
cpan
docs/content/docs/(concepts)/meta.json
cpan
docs/content/docs/(images)/meta.json
cpan
docs/content/docs/(multi-turn)/meta.json
cpan
docs/content/integrations/vector-databases/meta.json
cpan
docs/content/docs/(rag)/meta.json
cpan
docs/content/integrations/frameworks/meta.json
cpan
docs/content/docs/meta.json
cpan
docs/content/integrations/tts/meta.json
cpan
docs/content/integrations/voice-agent-connectors/meta.json
cpan
docs/content/integrations/stt/meta.json
cpan
.github/workflows/setup_wizard_test.yml
actions
- actions/checkout v4 composite
- actions/setup-node v4 composite
docs/content/docs/(behavioral-classifiers)/meta.json
cpan
docs/content/docs/(format-classifiers)/meta.json
cpan
docs/content/docs/(security-classifiers)/meta.json
cpan
docs/content/docs/(grounding-classifiers)/meta.json
cpan
docs/content/docs/(policy-classifiers)/meta.json
cpan
setup-wizard/package.json
npm
- @eslint/js latest development
- @types/node latest development
- eslint latest development
- postject latest development
- prettier latest development
- rolldown latest development
- typescript latest development
- typescript-eslint latest development
- vitest latest development
- @clack/prompts latest
- clipboardy latest
- open latest
- picocolors latest
- zod latest
setup-wizard/package-lock.json
npm
- 217 dependencies
.github/workflows/setup_wizard_release.yml
actions
- actions/checkout v4 composite
- actions/download-artifact v4 composite
- actions/setup-node v4 composite
- actions/upload-artifact v4 composite