data-juicer

data data-analysis data-pipeline data-processing data-science data-visualization foundation-models instruction-tuning large-language-models llm llms multi-modal pre-training synthetic-data

Score: 20.604243475142532

Last synced: about 4 hours ago
JSON representation

Repository metadata:

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

Host: GitHub
URL: https://github.com/datajuicer/data-juicer
Owner: datajuicer
License: apache-2.0
Created: 2023-08-01T09:16:41.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2026-10-27T03:17:14.000Z (about 15 hours ago)
Last Synced: 2026-10-27T09:41:32.420Z (about 8 hours ago)
Topics: data, data-analysis, data-pipeline, data-processing, data-science, data-visualization, foundation-models, instruction-tuning, large-language-models, llm, llms, multi-modal, pre-training, synthetic-data
Language: Python
Homepage: https://datajuicer.github.io/data-juicer/
Size: 227 MB
Stars: 7,189
Watchers: 20
Forks: 439
Open Issues: 72
Metadata Files:
- Readme: README.md
- License: LICENSE

Owner metadata:

Name: DataJuicer
Login: datajuicer
Email: datajuicer@outlook.com
Kind: organization
Description: Data processing for and with large models.
Website:
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/223222708?v=4
Repositories: 8
Last Synced at: 2026-10-24T21:24:51.813Z
Profile URL: https://github.com/datajuicer

GitHub Events

Total

Create event: 35
Delete event: 45
Discussion event: 1
Fork event: 25
Issue comment event: 78
Issues event: 32
Member event: 2
Pull request event: 101
Pull request review comment event: 154
Pull request review event: 102
Push event: 335
Watch event: 273
Total: 1183

Last Year

Create event: 35
Delete event: 45
Discussion event: 1
Fork event: 25
Issue comment event: 78
Issues event: 32
Member event: 2
Pull request event: 101
Pull request review comment event: 154
Pull request review event: 102
Push event: 335
Watch event: 273
Total: 1183

Committers metadata

Last synced: about 5 hours ago

Total Commits: 656
Total Committers: 69
Avg Commits per committer: 9.507
Development Distribution Score (DDS): 0.762

Commits in past year: 213
Committers in past year: 46
Avg Commits per committer in past year: 4.63
Development Distribution Score (DDS) in past year: 0.723

Name	Email	Commits
Yilun Huang	l**l@a**m	156
MeiXin Chen	8****n	76
Daoyuan Chen	6****c	51
BeachWang	1**7@p**n	48
Cathy0908	3****8	42
Ce Ge (戈策)	g**e@f**m	35
zhijianma	z**j@a**m	30
garyzhang99	4****9	17
Qirui-jiao	1****o	16
Cyrus Zhang	c**g@g**m	14
Fengrui Zhang	3****z	13
Xuchen Pan	3****c	12
chenhesen	h**s@a**m	12
Du Bin	d**5@g**m	11
Yuhan Liu	3****x	10
co63oc	c****c	10
Zhen Qin	z**n@g**m	7
John Giorgi	j**i@g**m	7
Xinyu Zhang	6****h	6
hongyu guo	g**7@b**m	6
chenyushuo	2**6@q**m	5
Mohammad Hijjawi	5****7	4
kyotom	3****m	4
林SO	1****n	3
lingzhq	1****q	3
Muneeb Ullah	m**5@g**m	3
HunterLine	1**0@q**m	3
Bin	1**2@q**m	3
2108038773	1****3	2
JamieYu	y**a@f**m	2
and 39 more...

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 20
Total pull requests: 136
Average time to close issues: about 2 months
Average time to close pull requests: 13 days
Total issue authors: 14
Total pull request authors: 32
Average comments per issue: 0.45
Average comments per pull request: 0.55
Merged pull request: 105
Bot issues: 0
Bot pull requests: 0

Past year issues: 20
Past year pull requests: 136
Past year average time to close issues: about 2 months
Past year average time to close pull requests: 13 days
Past year issue authors: 14
Past year pull request authors: 32
Past year average comments per issue: 0.45
Past year average comments per pull request: 0.55
Past year merged pull request: 105
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/datajuicer/data-juicer

Top Issue Authors

macroguo-ghy (5)
moneebullah25 (3)
fightBoxing (1)
cheramvb (1)
xww-uestc (1)
shaurya416 (1)
ArdalanM (1)
Mark-Wu2003 (1)
CodyQin (1)
gao-xiao-long (1)
cmgzn (1)
CoderChen01 (1)
qifengz (1)
JohnGiorgi (1)

Top Pull Request Authors

cmgzn (58)
fengrui-z (19)
Qirui-jiao (10)
macroguo-ghy (6)
MohammadHijjawi97 (5)
Cathy0908 (4)
Linxiushen (3)
xlb3055 (3)
vamsibvv (3)
moneebullah25 (3)
Dludora (2)
wenhaozhao011-cmd (2)
katsugtgz (2)
vertebrateqing (2)
prodmanpd (1)

Top Issue Labels

question (5)
enhancement (3)
good first issue (1)
environment (1)
dj:dataset (1)
dj:core (1)

Top Pull Request Labels

dj:op (6)
enhancement (6)
dj:multimodal (4)
ready-for-review (1)
dj:post-tuning (1)
dj:cookbook (1)
agent (1)

Package metadata

Total packages: 2
Total downloads:
- pypi: 1,771 last-month
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 0 (may contain duplicates)
Total versions: 64
Total maintainers: 1

proxy.golang.org: github.com/datajuicer/data-juicer

Homepage:
Documentation: https://pkg.go.dev/github.com/datajuicer/data-juicer#section-documentation
Licenses: apache-2.0
Latest release: v1.6.1 (published 7 days ago)
Last Synced: 2026-10-27T13:16:19.902Z (about 5 hours ago)
Versions: 31
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent packages count: 4.547%
- Average: 4.7%
- Dependent repos count: 4.853%

pypi.org: py-data-juicer

Data Processing for and with Foundation Models.

Homepage:
Documentation: https://py-data-juicer.readthedocs.io/
Licenses: Apache-2.0
Latest release: 1.6.1 (published 8 days ago)
Last Synced: 2026-10-27T13:32:06.354Z (about 4 hours ago)
Versions: 33
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,771 Last month
Rankings:
- Stargazers count: 7.067%
- Dependent packages count: 7.382%
- Forks count: 11.988%
- Downloads: 16.887%
- Average: 22.447%
- Dependent repos count: 68.91%
Maintainers (1)
- data-juicer

Dependencies

.github/workflows/perf-bench.yml actions

actions/checkout v3 composite

.github/workflows/pre-commit.yml actions

actions/checkout v4 composite
actions/setup-python v4 composite
pre-commit/action v3.0.0 composite

pyproject.toml pypi

.github/workflows/publish-docker-oss.yml actions

.github/workflows/unit-test-partial.yml actions

actions/checkout v3 composite
actions/download-artifact v4 composite
actions/upload-artifact v4 composite

demos/ego_hand_action_annotation/Dockerfile docker

pytorch/pytorch 2.4.1-cuda12.4-cudnn9-devel build

.github/workflows/unit-test.yml actions

actions/checkout v3 composite
actions/download-artifact v4 composite
actions/upload-artifact v4 composite
schneegans/dynamic-badges-action v1.7.0 composite

Dockerfile docker

nvidia/cuda 12.6.3-cudnn-devel-ubuntu24.04 build

.github/workflows/deploy_sphinx_docs.yml actions

actions/checkout v4 composite
actions/setup-python master composite
actions/upload-artifact v4 composite
astral-sh/setup-uv v7 composite
peaceiris/actions-gh-pages v3 composite

.github/workflows/publish-docker.yml actions

actions/checkout v3 composite
docker/build-push-action v6 composite
docker/login-action v2 composite
docker/metadata-action v5 composite
docker/setup-buildx-action v2 composite

.github/workflows/publish-pypi.yml actions

actions/checkout v4 composite
actions/setup-python v5 composite
pypa/gh-action-pypi-publish 27b31702a0e7fc50959f5ad993c78deac1bdfc29 composite
pypa/hatch install composite

.github/workflows/stale.yml actions

actions/stale v5 composite

.github/workflows/traffic-report.yml actions

actions/checkout v4 composite
actions/setup-python v5 composite

.github/workflows/docker/docker-compose.yml docker

data-juicer-unittest latest

docs/requirements-sphinx.txt pypi

IPython *
PyYAML *
diffusers ==0.35.2
docstring-parser *
einops *
fire *
linkify-it-py *
myst-parser ==5.1.0
onnxruntime *
python-dotenv *
pytorch-lightning ==2.6.5
setuptools *
smplx *
sphinx ==9.0.4
sphinx-copybutton ==0.5.2
transformers ==4.57.1
wandb >=0.19.11,<0.22
yacs *

uv.lock pypi

528 dependencies

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Summary