tokenizers

baichuan chatglm chatgpt gpt4 llama2 llama3 mistral tokenizer transfomers

Score: 16.127930133517175

Last synced: about 15 hours ago
JSON representation

Repository metadata:

a lightweight no-dependency fork from transformers.js (only tokenizers)

Host: GitHub
URL: https://github.com/lenML/tokenizers
Owner: lenML
License: mit
Created: 2024-04-20T09:11:18.000Z (over 2 years ago)
Default Branch: main
Last Pushed: 2026-05-16T11:25:47.000Z (3 months ago)
Last Synced: 2026-07-24T08:05:35.711Z (20 days ago)
Topics: baichuan, chatglm, chatgpt, gpt4, llama2, llama3, mistral, tokenizer, transfomers
Language: JavaScript
Homepage:
Size: 1.64 MB
Stars: 34
Watchers: 1
Forks: 1
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Owner metadata:

Name: lenML
Login: lenML
Email:
Kind: organization
Description:
Website:
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/154655365?v=4
Repositories: 1
Last Synced at: 2023-12-31T10:24:24.070Z
Profile URL: https://github.com/lenML

GitHub Events

Total

Fork event: 1
Issue comment event: 6
Issues event: 6
Push event: 13
Watch event: 12
Total: 38

Last Year

Push event: 3
Watch event: 3
Total: 6

Committers metadata

Last synced: 1 day ago

Total Commits: 72
Total Committers: 1
Avg Commits per committer: 72.0
Development Distribution Score (DDS): 0.0

Commits in past year: 11
Committers in past year: 1
Avg Commits per committer in past year: 11.0
Development Distribution Score (DDS) in past year: 0.0

Name	Email	Commits
zhzluke96	z**6@o**m	72

Issue and Pull Request metadata

Last synced: 2 months ago

Total issues: 10
Total pull requests: 0
Average time to close issues: 28 days
Average time to close pull requests: N/A
Total issue authors: 4
Total pull request authors: 0
Average comments per issue: 0.8
Average comments per pull request: 0
Merged pull request: 0
Bot issues: 0
Bot pull requests: 0

Past year issues: 2
Past year pull requests: 0
Past year average time to close issues: 7 months
Past year average time to close pull requests: N/A
Past year issue authors: 1
Past year pull request authors: 0
Past year average comments per issue: 1.0
Past year average comments per pull request: 0
Past year merged pull request: 0
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/lenML/tokenizers

Top Issue Authors

zcbenz (6)
zhzLuke96 (2)
huyiwen (1)
punkpeye (1)

Top Pull Request Authors

Top Issue Labels

enhancement (3)
bug (2)

Top Pull Request Labels

Package metadata

Total packages: 34
Total downloads:
- npm: 288,537 last-month
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 0 (may contain duplicates)
Total versions: 324
Total maintainers: 1

npmjs.org: @lenml/tokenizer-minicpm_v4_5

minicpm_v4_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.815Z (1 day ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 14 Last month
Rankings:
- Stargazers count: 6.444%
- Forks count: 9.361%
- Average: 18.545%
- Dependent repos count: 23.899%
- Dependent packages count: 34.476%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gptoss

gptoss tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.325Z (1 day ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 710 Last month
Rankings:
- Stargazers count: 6.444%
- Forks count: 9.361%
- Average: 18.545%
- Dependent repos count: 23.899%
- Dependent packages count: 34.476%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen3

qwen3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.372Z (1 day ago)
Versions: 2
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 3,672 Last month
Rankings:
- Stargazers count: 6.61%
- Forks count: 9.375%
- Average: 19.096%
- Dependent repos count: 24.718%
- Dependent packages count: 35.679%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen2_5

qwen2_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.132Z (1 day ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 111 Last month
Rankings:
- Downloads: 9.505%
- Average: 23.981%
- Dependent repos count: 25.506%
- Dependent packages count: 36.932%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-deepseek_v4

deepseek_v4 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 3 months ago)
Last Synced: 2026-08-11T17:01:03.622Z (1 day ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 54 Last month
Rankings:
- Dependent repos count: 21.678%
- Average: 26.462%
- Dependent packages count: 31.245%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-claude

claude tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.427Z (1 day ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 10,891 Last month
Rankings:
- Forks count: 14.087%
- Stargazers count: 20.118%
- Average: 27.123%
- Dependent repos count: 30.551%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-chatglm3

chatglm3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.518Z (1 day ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 162 Last month
Rankings:
- Forks count: 14.087%
- Stargazers count: 20.118%
- Average: 27.123%
- Dependent repos count: 30.551%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma3

gemma3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.421Z (1 day ago)
Versions: 3
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 55,768 Last month
Rankings:
- Dependent repos count: 24.827%
- Average: 30.365%
- Dependent packages count: 35.903%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-deepseek_v3

deepseek_v3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.655Z (1 day ago)
Versions: 4
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 6,718 Last month
Rankings:
- Dependent repos count: 24.868%
- Average: 30.415%
- Dependent packages count: 35.963%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemini

gemini tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.328Z (1 day ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 15,021 Last month
Rankings:
- Dependent repos count: 25.499%
- Average: 31.21%
- Dependent packages count: 36.921%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3_2

llama3_2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.576Z (1 day ago)
Versions: 4
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 168 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-aya_expanse

aya_expanse tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.468Z (1 day ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 52 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma2

gemma2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.156Z (1 day ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,063 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-mistral_nemo

mistral_nemo tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:02.887Z (1 day ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 2,278 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3_1

llama3_1 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.474Z (1 day ago)
Versions: 6
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,229 Last month
Rankings:
- Dependent repos count: 26.078%
- Average: 31.982%
- Dependent packages count: 37.885%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt4o

gpt4o tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.399Z (1 day ago)
Versions: 11
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 2,886 Last month
Rankings:
- Dependent repos count: 29.189%
- Average: 35.478%
- Dependent packages count: 41.767%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt35turbo

gpt35turbo tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.136Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,493 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-internlm2

internlm2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.451Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 165 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-baichuan2

baichuan2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:04.135Z (1 day ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 119 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_davinci003

text_davinci003 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.169Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 18 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma

gemma tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.413Z (1 day ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,034 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_davinci002

text_davinci002 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.197Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 100 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-command_r_plus

command_r_plus tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.182Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 82 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt4

gpt4 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.009Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 224 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_embedding_ada002

text_embedding_ada002 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.262Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 92 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-claude1

claude1 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.580Z (1 day ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 84 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt3

gpt3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.338Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 81 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizers

a lightweight no-dependency fork of transformers.js (only tokenizers)

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.408Z (1 day ago)
Versions: 15
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 177,285 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen1_5

qwen1_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.521Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 82 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt35turbo16k

gpt35turbo16k tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.317Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 74 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3

llama3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.275Z (1 day ago)
Versions: 12
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 3,150 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama2

llama2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.238Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,717 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-yi

yi tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.220Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,786 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt2

gpt2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 11 months ago)
Last Synced: 2026-08-11T17:01:03.207Z (1 day ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 154 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

Dependencies

package.json npm

semver ^7.6.0

packages/baichuan2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/chatglm3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/claude/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/claude1/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/command_r_plus/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gemma/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt35turbo/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt35turbo16k/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt4/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/internlm2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/llama2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/llama3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/qwen1_5/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_davinci002/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_davinci003/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_embedding_ada002/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/tokenizers/package.json npm

@huggingface/jinja ^0.2.2 development
@types/node ^20.12.7 development
@xenova/transformers ^2.17.1 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development

packages/tokenizers/pnpm-lock.yaml npm

244 dependencies

packages/yi/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

pnpm-lock.yaml npm

244 dependencies

scripts/pkg_template/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^