https://github.com/niieani/gpt-tokenizer
bpe decoder encoder gpt-2 gpt-3 gpt-4 gpt-4o gpt-5 gpt-o1 machine-learning openai tokenizer
Score: 26.615423062704988
Last synced: about 18 hours ago
JSON representation
Repository metadata:
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
- Host: GitHub
- URL: https://github.com/niieani/gpt-tokenizer
- Owner: niieani
- License: mit
- Created: 2023-03-22T04:32:09.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2026-08-16T04:03:34.000Z (9 days ago)
- Last Synced: 2026-08-20T09:38:36.707Z (5 days ago)
- Topics: bpe, decoder, encoder, gpt-2, gpt-3, gpt-4, gpt-4o, gpt-5, gpt-o1, machine-learning, openai, tokenizer
- Language: TypeScript
- Homepage: https://gpt-tokenizer.dev
- Size: 14.9 MB
- Stars: 835
- Watchers: 4
- Forks: 60
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Agents: AGENTS.md
-
Funding:
- Github: niieani
Owner metadata:
- Name: Bazyli Brzóska
- Login: niieani
- Email:
- Kind: user
- Description: An inventor passionate about HCI workflows that help humanity innovate responsibly, for happiness and global thriving. Official @zendesk account: @bbrzoska
- Website: https://invent.life
- Location: San Francisco, CA
- Twitter: niieani
- Company: @zendesk
- Icon url: https://avatars.githubusercontent.com/u/563469?v=4
- Repositories: 220
- Last Synced at: 2026-08-22T18:26:30.641Z
- Profile URL: https://github.com/niieani
GitHub Events
Total
- Create event: 13
- Delete event: 5
- Fork event: 11
- Issue comment event: 42
- Issues event: 22
- Pull request event: 17
- Pull request review comment event: 1
- Pull request review event: 1
- Push event: 34
- Release event: 10
- Watch event: 194
- Total: 350
Last Year
- Fork event: 3
- Issue comment event: 6
- Issues event: 4
- Pull request event: 2
- Pull request review comment event: 1
- Pull request review event: 1
- Push event: 10
- Watch event: 56
- Total: 83
Committers metadata
Last synced: 6 days ago
Total Commits: 143
Total Committers: 13
Avg Commits per committer: 11.0
Development Distribution Score (DDS): 0.336
Commits in past year: 32
Committers in past year: 2
Avg Commits per committer in past year: 16.0
Development Distribution Score (DDS) in past year: 0.031
| Name | Commits | |
|---|---|---|
| Bazyli Brzoska | b****a@g****m | 95 |
| Nick Walton | n****0@g****m | 18 |
| Anar Kafkas | a****s@g****m | 8 |
| mbukeRepo | m****4@g****m | 7 |
| renovate[bot] | 2****] | 4 |
| Squibs | Z****n@g****m | 3 |
| Sean Gillespie | i****o@s****s | 2 |
| Michael Sharpe | m****e@p****g | 1 |
| Justin Steven | j****n@j****m | 1 |
| David Schnurr | s****r@g****m | 1 |
| Caleb Bartholomew | c****b@a****o | 1 |
| Andrew Healey | d****y@g****m | 1 |
| Aman | 4****n | 1 |
Issue and Pull Request metadata
Last synced: 2 days ago
Total issues: 46
Total pull requests: 50
Average time to close issues: 4 months
Average time to close pull requests: 4 months
Total issue authors: 43
Total pull request authors: 9
Average comments per issue: 2.04
Average comments per pull request: 1.02
Merged pull request: 24
Bot issues: 2
Bot pull requests: 29
Past year issues: 3
Past year pull requests: 2
Past year average time to close issues: 3 months
Past year average time to close pull requests: 4 days
Past year issue authors: 3
Past year pull request authors: 2
Past year average comments per issue: 1.33
Past year average comments per pull request: 2.0
Past year merged pull request: 1
Past year bot issues: 1
Past year bot pull requests: 0
Top Issue Authors
- nilsreichardt (2)
- BennyAlex (2)
- thdoan (2)
- Qarj (1)
- aminsol (1)
- Ranork (1)
- renovate[bot] (1)
- emfastic (1)
- rafalzawadzki (1)
- Ferossgp (1)
- Zwe1 (1)
- enijar (1)
- gbrlmtrz (1)
- congqi09 (1)
- alejandro5042 (1)
Top Pull Request Authors
- renovate[bot] (29)
- niieani (10)
- mbukeRepo (4)
- shism2 (2)
- pete183 (1)
- Copilot (1)
- onlyoneaman (1)
- wesbos (1)
- justinsteven (1)
Top Issue Labels
- released (8)
- invalid (2)
- released on @beta (2)
- help wanted (2)
- documentation (1)
- semantic-release (1)
Top Pull Request Labels
- released (13)
Package metadata
- Total packages: 1
-
Total downloads:
- npm: 5,310,236 last-month
- Total docker downloads: 27,936,047
- Total dependent packages: 36
- Total dependent repositories: 22
- Total versions: 35
- Total maintainers: 1
npmjs.org: gpt-tokenizer
A pure JavaScript BPE tokenizer with model metadata and pricing for GPT-5, GPT-4o, o-series, and other OpenAI models
- Homepage: https://github.com/niieani/gpt-tokenizer#readme
- Licenses: MIT
- Latest release: 4.0.0 (published 9 days ago)
- Last Synced: 2026-08-23T11:03:35.289Z (2 days ago)
- Versions: 35
- Dependent Packages: 36
- Dependent Repositories: 22
- Downloads: 5,310,236 Last month
- Docker Downloads: 27,936,047
-
Rankings:
- Downloads: 0.518%
- Dependent packages count: 0.76%
- Docker downloads count: 1.874%
- Average: 2.41%
- Dependent repos count: 2.626%
- Stargazers count: 3.619%
- Forks count: 5.061%
- Maintainers (1)
Dependencies
- actions/checkout v3 composite
- actions/setup-node v3 composite
- actions/checkout v3 composite
- github/codeql-action/analyze v2 composite
- github/codeql-action/autobuild v2 composite
- github/codeql-action/init v2 composite
- @niieani/scaffold ^1.7.1 development
- tsx ^3.12.7 development
- rfc4648 ^1.5.2
- rfc4648 ^1.5.2
- 1385 dependencies
- actions/checkout v4 composite
- actions/configure-pages v5 composite
- actions/deploy-pages v4 composite
- actions/setup-node v4 composite
- actions/upload-pages-artifact v4 composite
- @types/node ^22.8.7 development
- chalk ^5.3.0 development
- cli-table3 ^0.6.5 development
- inquirer ^12.0.1 development
- typescript ^5.6.3 development
- gpt-3-encoder ^1.1.4
- gpt-tokenizer 2.2.3
- gpt3-tokenizer ^1.1.5
- js-tiktoken ^1.0.15
- tiktoken ^1.0.17
- tiktoken-node ^0.0.7
- @tailwindcss/vite ^4.1.14 development
- @types/react ^19.2.2 development
- @types/react-dom ^19.2.1 development
- @vitejs/plugin-react ^5.0.4 development
- autoprefixer ^10.4.21 development
- postcss ^8.5.6 development
- tailwindcss ^4.1.14 development
- typescript ^5.9.3 development
- vite ^7.1.9 development
- gpt-tokenizer ^3.3.0
- react ^19.2.0
- react-dom ^19.2.0
- 159 dependencies
- @colors/colors 1.5.0
- @inquirer/checkbox 4.0.1
- @inquirer/confirm 5.0.1
- @inquirer/core 10.0.1
- @inquirer/editor 4.0.1
- @inquirer/expand 4.0.1
- @inquirer/figures 1.0.7
- @inquirer/input 4.0.1
- @inquirer/number 3.0.1
- @inquirer/password 4.0.1
- @inquirer/prompts 7.0.1
- @inquirer/rawlist 4.0.1
- @inquirer/search 3.0.1
- @inquirer/select 4.0.1
- @inquirer/type 3.0.0
- @types/node 22.8.7
- ansi-escapes 4.3.2
- ansi-regex 5.0.1
- ansi-styles 4.3.0
- array-keyed-map 2.1.3
- base64-js 1.5.1
- chalk 5.3.0
- chardet 0.7.0
- cli-table3 0.6.5
- cli-width 4.1.0
- color-convert 2.0.1
- color-name 1.1.4
- emoji-regex 8.0.0
- external-editor 3.1.0
- gpt-3-encoder 1.1.4
- gpt-tokenizer 2.2.3
- gpt3-tokenizer 1.1.5
- iconv-lite 0.4.24
- inquirer 12.0.1
- is-fullwidth-code-point 3.0.0
- js-tiktoken 1.0.15
- mute-stream 2.0.0
- os-tmpdir 1.0.2
- rfc4648 1.5.3
- run-async 3.0.0
- rxjs 7.8.1
- safer-buffer 2.1.2
- signal-exit 4.1.0
- string-width 4.2.3
- strip-ansi 6.0.1
- tiktoken 1.0.17
- tiktoken-node 0.0.7
- tmp 0.0.33
- tslib 2.7.0
- type-fest 0.21.3
- typescript 5.6.3
- undici-types 6.19.8
- wrap-ansi 6.2.0
- yoctocolors-cjs 2.1.2