data-juicer

data data-analysis data-pipeline data-processing data-science data-visualization foundation-models instruction-tuning large-language-models llm llms multi-modal pre-training synthetic-data

Score: 20.773622088446373

Last synced: about 10 hours ago
JSON representation

Repository metadata:

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

Host: GitHub
URL: https://github.com/datajuicer/data-juicer
Owner: datajuicer
License: apache-2.0
Created: 2023-08-01T09:16:41.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2026-08-07T09:43:47.000Z (6 days ago)
Last Synced: 2026-08-08T07:10:08.209Z (5 days ago)
Topics: data, data-analysis, data-pipeline, data-processing, data-science, data-visualization, foundation-models, instruction-tuning, large-language-models, llm, llms, multi-modal, pre-training, synthetic-data
Language: Python
Homepage: https://datajuicer.github.io/data-juicer/
Size: 1.66 GB
Stars: 6,846
Watchers: 19
Forks: 404
Open Issues: 60
Metadata Files:
- Readme: README.md
- License: LICENSE

Owner metadata:

Name: DataJuicer
Login: datajuicer
Email: datajuicer@outlook.com
Kind: organization
Description: Data processing for and with large models.
Website:
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/223222708?v=4
Repositories: 1
Last Synced at: 2025-11-05T09:13:59.618Z
Profile URL: https://github.com/datajuicer

GitHub Events

Total

Create event: 34
Delete event: 43
Discussion event: 1
Fork event: 23
Issue comment event: 78
Issues event: 32
Member event: 2
Pull request event: 96
Pull request review comment event: 154
Pull request review event: 99
Push event: 325
Watch event: 263
Total: 1150

Last Year

Create event: 34
Delete event: 43
Discussion event: 1
Fork event: 23
Issue comment event: 78
Issues event: 32
Member event: 2
Pull request event: 96
Pull request review comment event: 154
Pull request review event: 99
Push event: 325
Watch event: 263
Total: 1150

Committers metadata

Last synced: 1 day ago

Total Commits: 605
Total Committers: 54
Avg Commits per committer: 11.204
Development Distribution Score (DDS): 0.742

Commits in past year: 194
Committers in past year: 31
Avg Commits per committer in past year: 6.258
Development Distribution Score (DDS) in past year: 0.742

Name	Email	Commits
Yilun Huang	l**l@a**m	156
MeiXin Chen	8****n	58
Daoyuan Chen	6****c	51
BeachWang	1**7@p**n	48
Cathy0908	3****8	42
Ce Ge (戈策)	g**e@f**m	35
zhijianma	z**j@a**m	30
garyzhang99	4****9	17
Qirui-jiao	1****o	15
Cyrus Zhang	c**g@g**m	14
Xuchen Pan	3****c	12
chenhesen	h**s@a**m	12
Du Bin	d**5@g**m	11
co63oc	c****c	10
Yuhan Liu	3****x	10
Fengrui Zhang	3****z	8
Zhen Qin	z**n@g**m	7
John Giorgi	j**i@g**m	6
Xinyu Zhang	6****h	6
hongyu guo	g**7@b**m	6
chenyushuo	2**6@q**m	5
kyotom	3****m	4
HunterLine	1**0@q**m	3
lingzhq	1****q	3
2108038773	1****3	2
JamieYu	y**a@f**m	2
ShenQianli	s**i@u**u	2
Yuexiang XIE	y**x@a**m	2
wenhaozhao011-cmd	w**1@g**m	2
Shurui Kou	1**2@q**m	2
and 24 more...

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 14
Total pull requests: 73
Average time to close issues: 2 months
Average time to close pull requests: 19 days
Total issue authors: 10
Total pull request authors: 16
Average comments per issue: 0.57
Average comments per pull request: 0.68
Merged pull request: 57
Bot issues: 0
Bot pull requests: 0

Past year issues: 14
Past year pull requests: 73
Past year average time to close issues: 2 months
Past year average time to close pull requests: 19 days
Past year issue authors: 10
Past year pull request authors: 16
Past year average comments per issue: 0.57
Past year average comments per pull request: 0.68
Past year merged pull request: 57
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/datajuicer/data-juicer

Top Issue Authors

macroguo-ghy (5)
fightBoxing (1)
gao-xiao-long (1)
CoderChen01 (1)
liumian576 (1)
qifengz (1)
xww-uestc (1)
ArdalanM (1)
Mark-Wu2003 (1)
CodyQin (1)

Top Pull Request Authors

cmgzn (34)
fengrui-z (10)
Qirui-jiao (7)
macroguo-ghy (6)
Cathy0908 (4)
wenhaozhao011-cmd (2)
yxdyc (1)
justinwolfington (1)
Dludora (1)
ideal (1)
LiuGuH (1)
ArdalanM (1)
CodyQin (1)
bglglzd (1)
nofear-spec (1)

Top Issue Labels

question (5)
enhancement (3)
good first issue (1)
environment (1)
dj:dataset (1)
dj:core (1)

Top Pull Request Labels

dj:op (6)
enhancement (5)
dj:multimodal (3)
agent (1)
dj:cookbook (1)
dj:post-tuning (1)

Package metadata

Total packages: 2
Total downloads:
- pypi: 2,819 last-month
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 0 (may contain duplicates)
Total versions: 60
Total maintainers: 1

proxy.golang.org: github.com/datajuicer/data-juicer

Homepage:
Documentation: https://pkg.go.dev/github.com/datajuicer/data-juicer#section-documentation
Licenses: apache-2.0
Latest release: v1.5.5 (published 6 days ago)
Last Synced: 2026-08-11T22:04:15.859Z (1 day ago)
Versions: 29
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent packages count: 4.547%
- Average: 4.7%
- Dependent repos count: 4.853%

pypi.org: py-data-juicer

Data Processing for and with Foundation Models.

Homepage:
Documentation: https://py-data-juicer.readthedocs.io/
Licenses: Apache-2.0
Latest release: 1.5.5 (published 6 days ago)
Last Synced: 2026-08-11T22:04:20.635Z (1 day ago)
Versions: 31
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 2,819 Last month
Rankings:
- Stargazers count: 7.067%
- Dependent packages count: 7.382%
- Forks count: 11.988%
- Downloads: 16.887%
- Average: 22.447%
- Dependent repos count: 68.91%
Maintainers (1)
- data-juicer

Dependencies

.github/workflows/perf-bench.yml actions

actions/checkout v3 composite

.github/workflows/pre-commit.yml actions

actions/checkout v4 composite
actions/setup-python v4 composite
pre-commit/action v3.0.0 composite

pyproject.toml pypi

.github/workflows/publish-docker-oss.yml actions

.github/workflows/unit-test-partial.yml actions

actions/checkout v3 composite
actions/download-artifact v4 composite
actions/upload-artifact v4 composite

demos/ego_hand_action_annotation/Dockerfile docker

pytorch/pytorch 2.4.1-cuda12.4-cudnn9-devel build

.github/workflows/unit-test.yml actions

actions/checkout v3 composite
actions/download-artifact v4 composite
actions/upload-artifact v4 composite
schneegans/dynamic-badges-action v1.7.0 composite

Dockerfile docker

nvidia/cuda 12.6.3-cudnn-devel-ubuntu24.04 build

.github/workflows/deploy_sphinx_docs.yml actions

actions/checkout v4 composite
actions/setup-python master composite
actions/upload-artifact v4 composite
astral-sh/setup-uv v7 composite
peaceiris/actions-gh-pages v3 composite

.github/workflows/publish-docker.yml actions

actions/checkout v3 composite
docker/build-push-action v6 composite
docker/login-action v2 composite
docker/metadata-action v5 composite
docker/setup-buildx-action v2 composite

.github/workflows/publish-pypi.yml actions

actions/checkout v4 composite
actions/setup-python v5 composite
pypa/gh-action-pypi-publish 27b31702a0e7fc50959f5ad993c78deac1bdfc29 composite
pypa/hatch install composite

.github/workflows/stale.yml actions

actions/stale v5 composite

.github/workflows/traffic-report.yml actions

actions/checkout v4 composite
actions/setup-python v5 composite

.github/workflows/docker/docker-compose.yml docker

data-juicer-unittest latest

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Summary