Data

Tools

Indexes

Applications

Experiments

Summary

An open API service for producing an overview of a list of open source projects.

https://github.com/huggingface/tokenizers

bert gpt language-model natural-language-processing natural-language-understanding nlp transformers

Score: 34.05683985576841

Last synced: about 9 hours ago
JSON representation

Repository metadata:

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Owner metadata:

GitHub Events

Total
Last Year

Committers metadata

Last synced: 5 days ago

Total Commits: 1,896
Total Committers: 150
Avg Commits per committer: 12.64
Development Distribution Score (DDS): 0.507

Commits in past year: 125
Committers in past year: 43
Avg Commits per committer in past year: 2.907
Development Distribution Score (DDS) in past year: 0.64

Name Email Commits
Anthony MOI m****i@g****m 934
Nicolas Patry p****s@p****m 271
Pierric Cistac p****c@h****o 138
Arthur 4****r 124
dependabot[bot] 4****] 53
epwalsh e****0@g****m 48
Morgan Funtowicz m****n@h****o 35
Sebastian Pütz s****z@u****e 28
Mishig Davaadorj d****g@g****m 23
Bjarte Johansen b****n@g****m 11
thomwolf t****f@g****m 8
Luc Georges M****e 8
Sylvain Gugger s****r@g****m 7
Chris Ha h****9@g****m 6
Nathan Goldbaum n****m@g****m 5
Simon Brandeis 3****s 5
sftse c****@f****t 5
Roy Hvaara h****a@g****m 5
Clement c****e@g****m 4
Connor Boyle c****o@g****m 4
Julien Chaumond c****d@g****m 4
Lysandre l****t@r****r 4
tinyboxvk t****k 3
Sebastian Pop s****p@n****m 3
大橋 玲音 1****n 3
hf-security-analysis[bot] 2****] 3
dctelus 9****s 3
Michael Feil 6****l 3
Lucain l****p@g****m 3
François Garillot f****s@g****t 3
and 120 more...

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 630
Total pull requests: 721
Average time to close issues: about 1 year
Average time to close pull requests: about 2 months
Total issue authors: 535
Total pull request authors: 165
Average comments per issue: 3.79
Average comments per pull request: 1.75
Merged pull request: 342
Bot issues: 0
Bot pull requests: 58

Past year issues: 29
Past year pull requests: 248
Past year average time to close issues: 27 days
Past year average time to close pull requests: 5 days
Past year issue authors: 28
Past year pull request authors: 59
Past year average comments per issue: 3.0
Past year average comments per pull request: 1.4
Past year merged pull request: 81
Past year bot issues: 0
Past year bot pull requests: 19

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/huggingface/tokenizers

Top Issue Authors

  • david-waterworth (15)
  • n1t0 (10)
  • Narsil (5)
  • SaulLu (4)
  • pietrolesci (4)
  • chris-ha458 (4)
  • davidgilbertson (4)
  • DOGEwbx (3)
  • EricLBuehler (3)
  • mullerhai (3)
  • jafioti (3)
  • xenova (3)
  • 8ria (3)
  • talolard (3)
  • DamonsJ (3)

Top Pull Request Authors

  • ArthurZucker (172)
  • Narsil (110)
  • dependabot[bot] (58)
  • SBrandeis (50)
  • McPatate (18)
  • sftse (10)
  • chris-ha458 (10)
  • sebpop (8)
  • hvaara (6)
  • ngoldbaum (6)
  • eaplatanios (5)
  • tinyboxvk (5)
  • boyleconnor (4)
  • bryantbiggs (4)
  • sondalex (4)

Top Issue Labels

  • Stale (316)
  • Feature Request (13)
  • bug (12)
  • enhancement (10)
  • planned (4)
  • good first issue (2)
  • bytefallback (1)
  • training (1)
  • bindings (1)
  • documentation (1)
  • python (1)
  • good second issue (1)

Top Pull Request Labels

  • Stale (60)
  • dependencies (58)
  • javascript (56)
  • target (8)
  • run-pipeline-bench (3)
  • github_actions (2)
  • Feature Request (1)

Package metadata

nixpkgs-23.05: python311Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-unstable: python314Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

pypi.org: tokenizers

  • Homepage: https://github.com/huggingface/tokenizers
  • Documentation: https://tokenizers.readthedocs.io/
  • Licenses: Apache Software License
  • Latest release: 0.23.1 (published 4 months ago)
  • Last Synced: 2026-08-24T14:03:58.890Z (1 day ago)
  • Versions: 123
  • Dependent Packages: 380
  • Dependent Repositories: 14,571
  • Downloads: 267,617,311 Last month
  • Docker Downloads: 42,285,347
  • Rankings:
    • Downloads: 0.057%
    • Dependent repos count: 0.068%
    • Dependent packages count: 0.086%
    • Docker downloads count: 0.599%
    • Stargazers count: 0.619%
    • Average: 0.626%
    • Forks count: 2.329%
  • Maintainers (5)
crates.io: tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

  • Homepage: https://github.com/huggingface/tokenizers
  • Documentation: https://docs.rs/tokenizers/
  • Licenses: Apache-2.0
  • Latest release: 0.23.1 (published 4 months ago)
  • Last Synced: 2026-08-24T14:04:12.559Z (1 day ago)
  • Versions: 40
  • Dependent Packages: 60
  • Dependent Repositories: 281
  • Downloads: 28,257,472 Total
  • Docker Downloads: 23,287,869
  • Rankings:
    • Stargazers count: 1.247%
    • Forks count: 1.477%
    • Dependent repos count: 2.361%
    • Average: 2.519%
    • Dependent packages count: 2.982%
    • Docker downloads count: 3.074%
    • Downloads: 3.97%
  • Maintainers (4)
npmjs.org: tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

  • Homepage: https://github.com/huggingface/tokenizers/tree/master/bindings/node
  • Licenses: Apache-2.0
  • Latest release: 0.13.3 (published over 3 years ago)
  • Last Synced: 2026-08-24T14:36:32.408Z (1 day ago)
  • Versions: 38
  • Dependent Packages: 6
  • Dependent Repositories: 23
  • Downloads: 57,961 Last month
  • Docker Downloads: 130
  • Rankings:
    • Stargazers count: 1.166%
    • Docker downloads count: 1.479%
    • Forks count: 1.502%
    • Dependent repos count: 2.673%
    • Average: 3.134%
    • Dependent packages count: 4.385%
    • Downloads: 7.598%
  • Maintainers (4)
proxy.golang.org: github.com/huggingface/tokenizers

  • Homepage:
  • Documentation: https://pkg.go.dev/github.com/huggingface/tokenizers#section-documentation
  • Licenses: apache-2.0
  • Latest release: v0.23.1 (published 4 months ago)
  • Last Synced: 2026-08-24T14:08:06.766Z (1 day ago)
  • Versions: 44
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Rankings:
    • Stargazers count: 0.809%
    • Forks count: 1.187%
    • Average: 3.794%
    • Dependent repos count: 4.802%
    • Dependent packages count: 8.376%
alpine-edge: py3-tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

  • Homepage: https://github.com/huggingface/tokenizers
  • Licenses: Apache-2.0
  • Latest release: 0.21.2-r1 (published 5 months ago)
  • Last Synced: 2026-03-30T15:47:10.014Z (5 months ago)
  • Versions: 17
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Rankings:
    • Dependent repos count: 0.0%
    • Stargazers count: 2.289%
    • Forks count: 4.25%
    • Average: 5.295%
    • Dependent packages count: 14.641%
  • Maintainers (1)
conda-forge.org: tokenizers

  • Homepage: https://pypi.org/project/tokenizers/
  • Licenses: Apache-2.0
  • Latest release: 0.13.1 (published almost 4 years ago)
  • Last Synced: 2026-04-01T16:16:59.926Z (5 months ago)
  • Versions: 16
  • Dependent Packages: 6
  • Dependent Repositories: 35
  • Downloads: 4,311,723 Total
  • Rankings:
    • Stargazers count: 4.188%
    • Dependent repos count: 6.114%
    • Average: 6.559%
    • Forks count: 6.898%
    • Dependent packages count: 9.034%
alpine-edge: py3-tokenizers-pyc

Precompiled Python bytecode for py3-tokenizers

  • Homepage: https://github.com/huggingface/tokenizers
  • Licenses: Apache-2.0
  • Latest release: 0.21.2-r1 (published 5 months ago)
  • Last Synced: 2026-03-30T15:47:10.721Z (5 months ago)
  • Versions: 16
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Rankings:
    • Dependent repos count: 0.0%
    • Average: 6.693%
    • Dependent packages count: 13.386%
  • Maintainers (1)
spack.io: py-tokenizers

Fast and Customizable Tokenizers.

  • Homepage: https://github.com/huggingface/tokenizers
  • Licenses: []
  • Latest release: 0.19.1 (published about 2 years ago)
  • Last Synced: 2026-08-24T14:05:11.770Z (1 day ago)
  • Versions: 7
  • Dependent Packages: 1
  • Dependent Repositories: 0
  • Rankings:
    • Dependent repos count: 0.0%
    • Stargazers count: 1.653%
    • Forks count: 3.946%
    • Average: 8.417%
    • Dependent packages count: 28.067%
  • Maintainers (1)
anaconda.org: tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

  • Homepage: https://github.com/huggingface/tokenizers
  • Licenses: Apache-2.0
  • Latest release: 0.23.1 (published 3 months ago)
  • Last Synced: 2026-05-20T14:49:45.224Z (3 months ago)
  • Versions: 12
  • Dependent Packages: 3
  • Dependent Repositories: 35
  • Downloads: 43,004 Total
  • Rankings:
    • Stargazers count: 9.936%
    • Forks count: 14.321%
    • Average: 23.106%
    • Dependent repos count: 27.231%
    • Dependent packages count: 40.938%
npmjs.org: @dkhokhlov/tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

  • Homepage: https://github.com/huggingface/tokenizers/tree/master/bindings/node
  • Licenses: Apache-2.0
  • Latest release: 0.22.1 (published 9 months ago)
  • Last Synced: 2026-08-24T14:22:38.543Z (1 day ago)
  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 16 Last month
  • Rankings:
    • Dependent repos count: 23.407%
    • Average: 28.584%
    • Dependent packages count: 33.761%
  • Maintainers (1)
pypi.org: divyanx-tokenizers

  • Homepage: https://github.com/huggingface/tokenizers
  • Documentation: https://divyanx-tokenizers.readthedocs.io/
  • Licenses: Apache Software License
  • Latest release: 0.20.0.dev0 (published about 2 years ago)
  • Last Synced: 2026-08-24T14:05:12.080Z (1 day ago)
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 13 Last month
  • Rankings:
    • Dependent packages count: 10.454%
    • Average: 34.65%
    • Dependent repos count: 58.847%
  • Maintainers (1)
pypi.org: tokenizers-gt

  • Homepage: https://github.com/huggingface/tokenizers
  • Documentation: https://tokenizers-gt.readthedocs.io/
  • Licenses: Apache Software License
  • Latest release: 0.15.2.post0 (published over 2 years ago)
  • Last Synced: 2026-08-24T14:04:07.388Z (1 day ago)
  • Versions: 3
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 1,045 Last month
  • Rankings:
    • Dependent packages count: 10.044%
    • Average: 38.708%
    • Dependent repos count: 67.371%
  • Maintainers (1)
npmjs.org: tokenizers-node

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

  • Homepage: https://github.com/huggingface/tokenizers/tree/master/bindings/node
  • Licenses: Apache-2.0
  • Latest release: 0.14.2-dev0 (published over 2 years ago)
  • Last Synced: 2026-08-24T14:28:37.574Z (1 day ago)
  • Versions: 3
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 21 Last month
  • Rankings:
    • Dependent repos count: 32.533%
    • Average: 39.609%
    • Dependent packages count: 46.685%
  • Maintainers (1)
gentoo-portage: sci-ml/tokenizers

Implementation of today's most used tokenizers

  • Homepage: https://github.com/huggingface/tokenizers
  • Documentation: https://packages.gentoo.org/packages/sci-ml/tokenizers
  • Licenses: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
  • Latest release: 0.23.1 (published 3 months ago)
  • Last Synced: 2026-07-24T01:47:40.869Z (about 1 month ago)
  • Versions: 3
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Rankings:
    • Dependent repos count: 0.0%
    • Dependent packages count: 0.0%
    • Average: 100%
nixpkgs-23.11: python310Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-24.11: python312Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-23.11: python311Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-24.05: python311Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-23.05: python310Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-24.05: python312Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-unstable: python313Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

nixpkgs-24.11: python311Packages.tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

Dependencies

.github/workflows/docs-check.yml actions
  • actions-rs/toolchain v1 composite
  • actions/checkout v1 composite
  • actions/setup-python v1 composite
  • actions/upload-artifact v2 composite
.github/workflows/node-release.yml actions
  • actions-rs/toolchain v1 composite
  • actions/cache v1 composite
  • actions/checkout v1 composite
  • actions/setup-node v1 composite
  • actions/setup-python v1 composite
.github/workflows/node.yml actions
  • actions-rs/cargo v1 composite
  • actions-rs/toolchain v1 composite
  • actions/cache v1 composite
  • actions/checkout v1 composite
  • actions/setup-node v1 composite
.github/workflows/python-release.yml actions
  • actions-rs/toolchain v1 composite
  • actions/checkout v2 composite
  • actions/checkout v1 composite
  • actions/setup-python v1 composite
  • actions/setup-python v4 composite
.github/workflows/python.yml actions
  • actions-rs/cargo v1 composite
  • actions-rs/toolchain v1 composite
  • actions/cache v1 composite
  • actions/checkout v1 composite
  • actions/setup-python v2 composite
.github/workflows/rust-release.yml actions
  • actions-rs/toolchain v1 composite
  • actions/cache v1 composite
  • actions/checkout v1 composite
.github/workflows/rust.yml actions
  • actions-rs/cargo v1 composite
  • actions-rs/toolchain v1 composite
  • actions/checkout v1 composite
bindings/python/Cargo.toml cargo
  • pyo3 0.17.2 development
  • tempfile 3.1 development
  • env_logger 0.7.1
  • itertools 0.9
  • libc 0.2
  • ndarray 0.13
  • numpy 0.17.2
  • onig 6.0
  • pyo3 0.17.2
  • rayon 1.3
  • serde 1.0
  • serde_json 1.0
  • tokenizers *
tokenizers/Cargo.toml cargo
  • assert_approx_eq 1.1 development
  • criterion 0.4 development
  • tempfile 3.1 development
  • aho-corasick 0.7
  • cached-path 0.6
  • clap 4.0
  • derive_builder 0.12
  • dirs 3.0
  • esaxx-rs 0.1
  • fancy-regex 0.10
  • getrandom 0.2.6
  • indicatif 0.15
  • itertools 0.9
  • lazy_static 1.4
  • log 0.4
  • macro_rules_attribute 0.1.2
  • onig 6.0
  • paste 1.0.6
  • rand 0.8
  • rayon 1.3
  • rayon-cond 0.1
  • regex 1.3
  • regex-syntax 0.6
  • reqwest 0.11
  • serde 1.0
  • serde_json 1.0
  • spm_precompiled 0.1
  • thiserror 1.0.30
  • unicode-normalization-alignments 0.1
  • unicode-segmentation 1.6
  • unicode_categories 0.1
bindings/node/package.json npm
  • @types/jest ^26.0.24 development
  • @typescript-eslint/eslint-plugin ^3.10.1 development
  • @typescript-eslint/parser ^3.10.1 development
  • eslint ^7.32.0 development
  • eslint-config-prettier ^6.15.0 development
  • eslint-plugin-jest ^23.20.0 development
  • eslint-plugin-jsdoc ^30.7.13 development
  • eslint-plugin-prettier ^3.4.1 development
  • eslint-plugin-simple-import-sort ^5.0.3 development
  • jest ^26.6.3 development
  • neon-cli ^0.9.1 development
  • prettier ^2.5.1 development
  • shelljs ^0.8.3 development
  • ts-jest ^26.5.6 development
  • typescript ^3.9.10 development
  • @types/node ^13.13.52
  • node-pre-gyp ^0.14.0
.github/workflows/delete_doc_comment.yml actions
bindings/node/npm/darwin-x64/package.json npm
bindings/node/npm/win32-x64-msvc/package.json npm
.github/workflows/trufflehog.yml actions
  • actions/checkout de0fac2e4500dabe0009e67214ff5f5447ce83dd composite
  • trufflesecurity/trufflehog 853e1e8d249fd1e29d0fcc7280d29b03df3d643d composite
bindings/node/npm/android-arm-eabi/package.json npm
.github/workflows/delete_doc_comment_trigger.yml actions
.github/workflows/upload_pr_documentation.yml actions
bindings/node/npm/linux-x64-musl/package.json npm
.github/workflows/stale.yml actions
  • actions/stale v9 composite
bindings/node/Cargo.toml cargo
bindings/node/npm/linux-arm64-musl/package.json npm
bindings/python/tools/stub-gen/Cargo.toml cargo
bindings/node/npm/android-arm64/package.json npm
bindings/node/npm/darwin-arm64/package.json npm
bindings/node/npm/freebsd-x64/package.json npm
.github/workflows/build_pr_documentation.yml actions
bindings/node/npm/linux-arm64-gnu/package.json npm
.github/workflows/docs_backfill.yml actions
bindings/node/npm/linux-x64-gnu/package.json npm
bindings/node/npm/win32-ia32-msvc/package.json npm
.github/workflows/benchmark-trigger.yml actions
.github/workflows/CI.yml actions
  • PyO3/maturin-action 04ac600d27cdf7a9a280dadf7147097c42b757ad composite
  • actions/attest-build-provenance ef244123eb79f2f7a7e75d99086184180e6d0018 composite
  • actions/checkout de0fac2e4500dabe0009e67214ff5f5447ce83dd composite
  • actions/download-artifact d3f86a106a0bac45b974a628896c90dbdf5c8093 composite
  • actions/setup-python a26af69be951a213d495a4c3e4e4022e16d87065 composite
  • actions/upload-artifact ea165f8d65b6e75b540449e92b4886f43607fa02 composite
.github/workflows/benchmarks.yml actions
  • actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
  • actions/upload-artifact 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a composite
  • astral-sh/setup-uv v6 composite
  • dtolnay/rust-toolchain stable composite
  • mozilla-actions/sccache-action v0.0.9 composite
bindings/node/npm/win32-arm64-msvc/package.json npm
bindings/node/yarn.lock npm
  • 784 dependencies
.github/workflows/build_documentation.yml actions
bindings/node/npm/linux-arm-gnueabihf/package.json npm
bindings/python/pyproject.toml pypi
  • huggingface-hub >=0.16.4,<2.0