tokenizers

baichuan chatglm chatgpt gpt4 llama2 llama3 mistral tokenizer transfomers

Score: 16.331252702765283

Last synced: about 21 hours ago
JSON representation

Repository metadata:

a lightweight no-dependency fork from transformers.js (only tokenizers)

Host: GitHub
URL: https://github.com/lenML/tokenizers
Owner: lenML
License: mit
Created: 2024-04-20T09:11:18.000Z (over 2 years ago)
Default Branch: main
Last Pushed: 2026-05-16T11:25:47.000Z (5 months ago)
Last Synced: 2026-07-24T08:05:35.711Z (2 months ago)
Topics: baichuan, chatglm, chatgpt, gpt4, llama2, llama3, mistral, tokenizer, transfomers
Language: JavaScript
Homepage:
Size: 1.64 MB
Stars: 34
Watchers: 1
Forks: 1
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Owner metadata:

Name: lenML
Login: lenML
Email: len-ml@outlook.com
Kind: organization
Description:
Website:
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/154655365?v=4
Repositories: 1
Last Synced at: 2026-10-02T20:39:34.655Z
Profile URL: https://github.com/lenML

GitHub Events

Total

Fork event: 1
Issue comment event: 6
Issues event: 6
Push event: 13
Watch event: 12
Total: 38

Last Year

Push event: 2
Watch event: 3
Total: 5

Committers metadata

Last synced: 2 days ago

Total Commits: 72
Total Committers: 1
Avg Commits per committer: 72.0
Development Distribution Score (DDS): 0.0

Commits in past year: 5
Committers in past year: 1
Avg Commits per committer in past year: 5.0
Development Distribution Score (DDS) in past year: 0.0

Name	Email	Commits
zhzluke96	z**6@o**m	72

Issue and Pull Request metadata

Last synced: 2 days ago

Total issues: 10
Total pull requests: 0
Average time to close issues: 28 days
Average time to close pull requests: N/A
Total issue authors: 4
Total pull request authors: 0
Average comments per issue: 0.8
Average comments per pull request: 0
Merged pull request: 0
Bot issues: 0
Bot pull requests: 0

Past year issues: 2
Past year pull requests: 0
Past year average time to close issues: 7 months
Past year average time to close pull requests: N/A
Past year issue authors: 1
Past year pull request authors: 0
Past year average comments per issue: 1.0
Past year average comments per pull request: 0
Past year merged pull request: 0
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/lenML/tokenizers

Top Issue Authors

zcbenz (6)
zhzLuke96 (2)
huyiwen (1)
punkpeye (1)

Top Pull Request Authors

Top Issue Labels

enhancement (3)
bug (2)

Top Pull Request Labels

Package metadata

Total packages: 34
Total downloads:
- npm: 353,593 last-month
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 0 (may contain duplicates)
Total versions: 324
Total maintainers: 1

npmjs.org: @lenml/tokenizer-gptoss

gptoss tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:53.902Z (2 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 363 Last month
Rankings:
- Stargazers count: 6.444%
- Forks count: 9.361%
- Average: 18.545%
- Dependent repos count: 23.899%
- Dependent packages count: 34.476%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-minicpm_v4_5

minicpm_v4_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:51.166Z (2 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 17 Last month
Rankings:
- Stargazers count: 6.444%
- Forks count: 9.361%
- Average: 18.545%
- Dependent repos count: 23.899%
- Dependent packages count: 34.476%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen3

qwen3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:05.813Z (2 days ago)
Versions: 2
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 3,018 Last month
Rankings:
- Stargazers count: 6.61%
- Forks count: 9.375%
- Average: 19.096%
- Dependent repos count: 24.718%
- Dependent packages count: 35.679%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen2_5

qwen2_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:12.050Z (2 days ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 148 Last month
Rankings:
- Downloads: 9.505%
- Average: 23.981%
- Dependent repos count: 25.506%
- Dependent packages count: 36.932%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-deepseek_v4

deepseek_v4 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2026-10-02T14:01:27.245Z (2 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 2,058 Last month
Rankings:
- Dependent repos count: 21.678%
- Average: 26.462%
- Dependent packages count: 31.245%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-claude

claude tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:51.424Z (2 days ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 11,757 Last month
Rankings:
- Forks count: 14.087%
- Stargazers count: 20.118%
- Average: 27.123%
- Dependent repos count: 30.551%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-chatglm3

chatglm3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:48.972Z (2 days ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 100 Last month
Rankings:
- Forks count: 14.087%
- Stargazers count: 20.118%
- Average: 27.123%
- Dependent repos count: 30.551%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma3

gemma3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:33.987Z (2 days ago)
Versions: 3
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 61,594 Last month
Rankings:
- Dependent repos count: 24.827%
- Average: 30.365%
- Dependent packages count: 35.903%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-deepseek_v3

deepseek_v3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:10.178Z (2 days ago)
Versions: 4
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 7,563 Last month
Rankings:
- Dependent repos count: 24.868%
- Average: 30.415%
- Dependent packages count: 35.963%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemini

gemini tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:50.890Z (2 days ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 14,482 Last month
Rankings:
- Dependent repos count: 25.499%
- Average: 31.21%
- Dependent packages count: 36.921%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-aya_expanse

aya_expanse tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:28.885Z (2 days ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 21 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-mistral_nemo

mistral_nemo tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:41.933Z (2 days ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 5,418 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3_2

llama3_2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:48.018Z (2 days ago)
Versions: 4
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 93 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma2

gemma2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:50.021Z (2 days ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 3,549 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3_1

llama3_1 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:59.782Z (2 days ago)
Versions: 6
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 372 Last month
Rankings:
- Dependent repos count: 26.078%
- Average: 31.982%
- Dependent packages count: 37.885%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt4o

gpt4o tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:08.507Z (2 days ago)
Versions: 11
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 286 Last month
Rankings:
- Dependent repos count: 29.189%
- Average: 35.478%
- Dependent packages count: 41.767%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma

gemma tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:13.915Z (2 days ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 401 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_davinci002

text_davinci002 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:11.484Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 28 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-internlm2

internlm2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:17.072Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 27 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizers

a lightweight no-dependency fork of transformers.js (only tokenizers)

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:19.072Z (2 days ago)
Versions: 15
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 238,008 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt2

gpt2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:20.763Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 88 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt35turbo16k

gpt35turbo16k tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:21.324Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 24 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt3

gpt3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:21.158Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 25 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-yi

yi tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:21.416Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 93 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen1_5

qwen1_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:22.086Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 58 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-command_r_plus

command_r_plus tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:02.050Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 18 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-claude1

claude1 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:58.294Z (2 days ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 31 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt4

gpt4 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:52.546Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 382 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-baichuan2

baichuan2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:50.380Z (2 days ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 32 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_davinci003

text_davinci003 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:31.804Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 29 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt35turbo

gpt35turbo tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:27.828Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 367 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3

llama3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:17.387Z (2 days ago)
Versions: 12
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,350 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_embedding_ada002

text_embedding_ada002 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:02:15.953Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 53 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama2

llama2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published about 1 year ago)
Last Synced: 2026-10-02T14:01:45.240Z (2 days ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,740 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

Dependencies

package.json npm

semver ^7.6.0

packages/baichuan2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/chatglm3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/claude/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/claude1/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/command_r_plus/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gemma/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt35turbo/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt35turbo16k/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt4/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/internlm2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/llama2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/llama3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/qwen1_5/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_davinci002/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_davinci003/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_embedding_ada002/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/tokenizers/package.json npm

@huggingface/jinja ^0.2.2 development
@types/node ^20.12.7 development
@xenova/transformers ^2.17.1 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development

packages/tokenizers/pnpm-lock.yaml npm

244 dependencies

packages/yi/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

pnpm-lock.yaml npm

244 dependencies

scripts/pkg_template/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^