https://github.com/niieani/gpt-tokenizer
bpe decoder encoder gpt-2 gpt-3 gpt-4 gpt-4o gpt-5 gpt-o1 machine-learning openai tokenizer
Score: 26.411045617582435
Last synced: about 15 hours ago
JSON representation
Repository metadata:
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
- Host: GitHub
- URL: https://github.com/niieani/gpt-tokenizer
- Owner: niieani
- License: mit
- Created: 2023-03-22T04:32:09.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2026-02-10T17:50:16.000Z (about 1 month ago)
- Last Synced: 2026-03-21T18:51:51.479Z (5 days ago)
- Topics: bpe, decoder, encoder, gpt-2, gpt-3, gpt-4, gpt-4o, gpt-5, gpt-o1, machine-learning, openai, tokenizer
- Language: TypeScript
- Homepage: https://gpt-tokenizer.dev
- Size: 13 MB
- Stars: 756
- Watchers: 3
- Forks: 55
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Agents: AGENTS.md
-
Funding:
- Github: niieani
Owner metadata:
- Name: Bazyli Brzóska
- Login: niieani
- Email:
- Kind: user
- Description: An inventor passionate about HCI workflows that help humanity innovate responsibly, for happiness and global thriving. Official @zendesk account: @bbrzoska
- Website: https://invent.life
- Location: San Francisco, CA
- Twitter: niieani
- Company: @zendesk
- Icon url: https://avatars.githubusercontent.com/u/563469?v=4
- Repositories: 206
- Last Synced at: 2026-01-25T21:32:44.093Z
- Profile URL: https://github.com/niieani
GitHub Events
Total
- Create event: 13
- Delete event: 5
- Fork event: 10
- Issue comment event: 42
- Issues event: 22
- Pull request event: 17
- Pull request review comment event: 1
- Pull request review event: 1
- Push event: 33
- Release event: 10
- Watch event: 188
- Total: 342
Last Year
- Create event: 3
- Fork event: 6
- Issue comment event: 15
- Issues event: 6
- Pull request event: 5
- Pull request review comment event: 1
- Pull request review event: 1
- Push event: 15
- Release event: 2
- Watch event: 102
- Total: 156
Committers metadata
Last synced: 6 days ago
Total Commits: 132
Total Committers: 13
Avg Commits per committer: 10.154
Development Distribution Score (DDS): 0.364
Commits in past year: 24
Committers in past year: 2
Avg Commits per committer in past year: 12.0
Development Distribution Score (DDS) in past year: 0.042
| Name | Commits | |
|---|---|---|
| Bazyli Brzoska | b****a@g****m | 84 |
| Nick Walton | n****0@g****m | 18 |
| Anar Kafkas | a****s@g****m | 8 |
| mbukeRepo | m****4@g****m | 7 |
| renovate[bot] | 2****] | 4 |
| Squibs | Z****n@g****m | 3 |
| Sean Gillespie | i****o@s****s | 2 |
| Michael Sharpe | m****e@p****g | 1 |
| Justin Steven | j****n@j****m | 1 |
| David Schnurr | s****r@g****m | 1 |
| Caleb Bartholomew | c****b@a****o | 1 |
| Andrew Healey | d****y@g****m | 1 |
| Aman | 4****n | 1 |
Issue and Pull Request metadata
Last synced: about 1 month ago
Total issues: 46
Total pull requests: 49
Average time to close issues: 4 months
Average time to close pull requests: 4 months
Total issue authors: 43
Total pull request authors: 8
Average comments per issue: 2.02
Average comments per pull request: 1.0
Merged pull request: 24
Bot issues: 2
Bot pull requests: 29
Past year issues: 5
Past year pull requests: 4
Past year average time to close issues: about 1 month
Past year average time to close pull requests: 1 day
Past year issue authors: 5
Past year pull request authors: 3
Past year average comments per issue: 0.6
Past year average comments per pull request: 0.75
Past year merged pull request: 3
Past year bot issues: 1
Past year bot pull requests: 0
Top Issue Authors
- nilsreichardt (2)
- BennyAlex (2)
- thdoan (2)
- Qarj (1)
- aminsol (1)
- Ranork (1)
- renovate[bot] (1)
- emfastic (1)
- rafalzawadzki (1)
- Ferossgp (1)
- Zwe1 (1)
- enijar (1)
- gbrlmtrz (1)
- congqi09 (1)
- alejandro5042 (1)
Top Pull Request Authors
- renovate[bot] (29)
- niieani (10)
- mbukeRepo (4)
- shism2 (2)
- Copilot (1)
- onlyoneaman (1)
- wesbos (1)
- justinsteven (1)
Top Issue Labels
- released (7)
- released on @beta (2)
- help wanted (2)
- invalid (2)
- documentation (1)
- semantic-release (1)
Top Pull Request Labels
- released (12)
Package metadata
- Total packages: 1
-
Total downloads:
- npm: 1,946,212 last-month
- Total docker downloads: 27,936,047
- Total dependent packages: 36
- Total dependent repositories: 22
- Total versions: 34
- Total maintainers: 1
npmjs.org: gpt-tokenizer
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
- Homepage: https://github.com/niieani/gpt-tokenizer#readme
- Licenses: MIT
- Latest release: 3.4.0 (published 5 months ago)
- Last Synced: 2026-03-21T05:06:38.991Z (6 days ago)
- Versions: 34
- Dependent Packages: 36
- Dependent Repositories: 22
- Downloads: 1,946,212 Last month
- Docker Downloads: 27,936,047
-
Rankings:
- Downloads: 0.518%
- Dependent packages count: 0.76%
- Docker downloads count: 1.874%
- Average: 2.41%
- Dependent repos count: 2.626%
- Stargazers count: 3.619%
- Forks count: 5.061%
- Maintainers (1)
Dependencies
- actions/checkout v3 composite
- actions/setup-node v3 composite
- actions/checkout v3 composite
- github/codeql-action/analyze v2 composite
- github/codeql-action/autobuild v2 composite
- github/codeql-action/init v2 composite
- @niieani/scaffold ^1.7.1 development
- tsx ^3.12.7 development
- rfc4648 ^1.5.2
- rfc4648 ^1.5.2
- 1385 dependencies
- actions/checkout v4 composite
- actions/configure-pages v5 composite
- actions/deploy-pages v4 composite
- actions/setup-node v4 composite
- actions/upload-pages-artifact v4 composite
- @types/node ^22.8.7 development
- chalk ^5.3.0 development
- cli-table3 ^0.6.5 development
- inquirer ^12.0.1 development
- typescript ^5.6.3 development
- gpt-3-encoder ^1.1.4
- gpt-tokenizer 2.2.3
- gpt3-tokenizer ^1.1.5
- js-tiktoken ^1.0.15
- tiktoken ^1.0.17
- tiktoken-node ^0.0.7
- @tailwindcss/vite ^4.1.14 development
- @types/react ^19.2.2 development
- @types/react-dom ^19.2.1 development
- @vitejs/plugin-react ^5.0.4 development
- autoprefixer ^10.4.21 development
- postcss ^8.5.6 development
- tailwindcss ^4.1.14 development
- typescript ^5.9.3 development
- vite ^7.1.9 development
- gpt-tokenizer ^3.3.0
- react ^19.2.0
- react-dom ^19.2.0
- 159 dependencies
- @colors/colors 1.5.0
- @inquirer/checkbox 4.0.1
- @inquirer/confirm 5.0.1
- @inquirer/core 10.0.1
- @inquirer/editor 4.0.1
- @inquirer/expand 4.0.1
- @inquirer/figures 1.0.7
- @inquirer/input 4.0.1
- @inquirer/number 3.0.1
- @inquirer/password 4.0.1
- @inquirer/prompts 7.0.1
- @inquirer/rawlist 4.0.1
- @inquirer/search 3.0.1
- @inquirer/select 4.0.1
- @inquirer/type 3.0.0
- @types/node 22.8.7
- ansi-escapes 4.3.2
- ansi-regex 5.0.1
- ansi-styles 4.3.0
- array-keyed-map 2.1.3
- base64-js 1.5.1
- chalk 5.3.0
- chardet 0.7.0
- cli-table3 0.6.5
- cli-width 4.1.0
- color-convert 2.0.1
- color-name 1.1.4
- emoji-regex 8.0.0
- external-editor 3.1.0
- gpt-3-encoder 1.1.4
- gpt-tokenizer 2.2.3
- gpt3-tokenizer 1.1.5
- iconv-lite 0.4.24
- inquirer 12.0.1
- is-fullwidth-code-point 3.0.0
- js-tiktoken 1.0.15
- mute-stream 2.0.0
- os-tmpdir 1.0.2
- rfc4648 1.5.3
- run-async 3.0.0
- rxjs 7.8.1
- safer-buffer 2.1.2
- signal-exit 4.1.0
- string-width 4.2.3
- strip-ansi 6.0.1
- tiktoken 1.0.17
- tiktoken-node 0.0.7
- tmp 0.0.33
- tslib 2.7.0
- type-fest 0.21.3
- typescript 5.6.3
- undici-types 6.19.8
- wrap-ansi 6.2.0
- yoctocolors-cjs 2.1.2