tokenizers

baichuan chatglm chatgpt gpt4 llama2 llama3 mistral tokenizer transfomers

Score: 14.208680387978287

Last synced: about 19 hours ago
JSON representation

Repository metadata:

a lightweight no-dependency fork from transformers.js (only tokenizers)

Host: GitHub
URL: https://github.com/lenML/tokenizers
Owner: lenML
License: mit
Created: 2024-04-20T09:11:18.000Z (almost 2 years ago)
Default Branch: main
Last Pushed: 2025-09-06T06:21:55.000Z (5 months ago)
Last Synced: 2025-12-06T15:45:43.541Z (2 months ago)
Topics: baichuan, chatglm, chatgpt, gpt4, llama2, llama3, mistral, tokenizer, transfomers
Language: JavaScript
Homepage:
Size: 1.54 MB
Stars: 29
Watchers: 1
Forks: 1
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Owner metadata:

Name: lenML
Login: lenML
Email:
Kind: organization
Description:
Website:
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/154655365?v=4
Repositories: 1
Last Synced at: 2023-12-31T10:24:24.070Z
Profile URL: https://github.com/lenML

GitHub Events

Total

Fork event: 1
Issue comment event: 6
Issues event: 6
Push event: 11
Watch event: 11
Total: 35

Last Year

Fork event: 1
Issues event: 1
Push event: 6
Watch event: 10
Total: 18

Committers metadata

Last synced: 2 months ago

Total Commits: 67
Total Committers: 1
Avg Commits per committer: 67.0
Development Distribution Score (DDS): 0.0

Commits in past year: 15
Committers in past year: 1
Avg Commits per committer in past year: 15.0
Development Distribution Score (DDS) in past year: 0.0

Name	Email	Commits
zhzluke96	z**6@o**m	67

Issue and Pull Request metadata

Last synced: 3 months ago

Total issues: 9
Total pull requests: 0
Average time to close issues: about 9 hours
Average time to close pull requests: N/A
Total issue authors: 4
Total pull request authors: 0
Average comments per issue: 0.67
Average comments per pull request: 0
Merged pull request: 0
Bot issues: 0
Bot pull requests: 0

Past year issues: 2
Past year pull requests: 0
Past year average time to close issues: N/A
Past year average time to close pull requests: N/A
Past year issue authors: 2
Past year pull request authors: 0
Past year average comments per issue: 0.0
Past year average comments per pull request: 0
Past year merged pull request: 0
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/lenML/tokenizers

Top Issue Authors

zcbenz (6)
huyiwen (1)
zhzLuke96 (1)
punkpeye (1)

Top Pull Request Authors

Top Issue Labels

enhancement (2)
bug (2)

Top Pull Request Labels

Package metadata

Total packages: 33
Total downloads:
- npm: 49,388 last-month
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 0 (may contain duplicates)
Total versions: 323
Total maintainers: 1

npmjs.org: @lenml/tokenizer-gptoss

gptoss tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:42.223Z (2 months ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 137 Last month
Rankings:
- Stargazers count: 6.444%
- Forks count: 9.361%
- Average: 18.545%
- Dependent repos count: 23.899%
- Dependent packages count: 34.476%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-minicpm_v4_5

minicpm_v4_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:35.340Z (2 months ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 15 Last month
Rankings:
- Stargazers count: 6.444%
- Forks count: 9.361%
- Average: 18.545%
- Dependent repos count: 23.899%
- Dependent packages count: 34.476%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen3

qwen3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:47.163Z (2 months ago)
Versions: 2
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,256 Last month
Rankings:
- Stargazers count: 6.61%
- Forks count: 9.375%
- Average: 19.096%
- Dependent repos count: 24.718%
- Dependent packages count: 35.679%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen2_5

qwen2_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:03:47.565Z (2 months ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 626 Last month
Rankings:
- Downloads: 9.505%
- Average: 23.981%
- Dependent repos count: 25.506%
- Dependent packages count: 36.932%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-claude

claude tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:47.153Z (3 months ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 4,696 Last month
Rankings:
- Forks count: 14.087%
- Stargazers count: 20.118%
- Average: 27.123%
- Dependent repos count: 30.551%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-chatglm3

chatglm3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:37.371Z (2 months ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 16 Last month
Rankings:
- Forks count: 14.087%
- Stargazers count: 20.118%
- Average: 27.123%
- Dependent repos count: 30.551%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma3

gemma3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:42.951Z (2 months ago)
Versions: 3
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 5,964 Last month
Rankings:
- Dependent repos count: 24.827%
- Average: 30.365%
- Dependent packages count: 35.903%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-deepseek_v3

deepseek_v3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:32.752Z (2 months ago)
Versions: 4
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 2,024 Last month
Rankings:
- Dependent repos count: 24.868%
- Average: 30.415%
- Dependent packages count: 35.963%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemini

gemini tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:47.125Z (2 months ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 2,509 Last month
Rankings:
- Dependent repos count: 25.499%
- Average: 31.21%
- Dependent packages count: 36.921%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-mistral_nemo

mistral_nemo tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:35.602Z (2 months ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 22 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-aya_expanse

aya_expanse tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:42.760Z (2 months ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 13 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3_2

llama3_2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:42.987Z (2 months ago)
Versions: 4
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 131 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma2

gemma2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:36.482Z (2 months ago)
Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 275 Last month
Rankings:
- Dependent repos count: 25.515%
- Average: 31.231%
- Dependent packages count: 36.947%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3_1

llama3_1 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:43.012Z (2 months ago)
Versions: 6
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 108 Last month
Rankings:
- Dependent repos count: 26.078%
- Average: 31.982%
- Dependent packages count: 37.885%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt4o

gpt4o tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:47.145Z (2 months ago)
Versions: 11
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 446 Last month
Rankings:
- Dependent repos count: 29.189%
- Average: 35.478%
- Dependent packages count: 41.767%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt35turbo

gpt35turbo tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:44.039Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 453 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_embedding_ada002

text_embedding_ada002 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:46.735Z (3 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 126 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_davinci003

text_davinci003 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:47.187Z (3 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 76 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt2

gpt2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:35.466Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 121 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-command_r_plus

command_r_plus tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:47.036Z (3 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 154 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-text_davinci002

text_davinci002 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:47.561Z (3 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 71 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-baichuan2

baichuan2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:47.452Z (3 months ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 97 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt3

gpt3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-09T02:03:47.818Z (3 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 97 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt4

gpt4 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:43.384Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 204 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-qwen1_5

qwen1_5 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:40.075Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 479 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-yi

yi tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:15.075Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 96 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama2

llama2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:03:53.524Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 903 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gpt35turbo16k

gpt35turbo16k tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:59.648Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 44 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-internlm2

internlm2 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:42.709Z (2 months ago)
Versions: 13
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 131 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-claude1

claude1 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:36.132Z (2 months ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 39 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizers

a lightweight no-dependency fork of transformers.js (only tokenizers)

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:05:35.676Z (2 months ago)
Versions: 15
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 25,975 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-gemma

gemma tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:28.410Z (2 months ago)
Versions: 14
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 96 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

npmjs.org: @lenml/tokenizer-llama3

llama3 tokenizer for NodeJS/Browser

Homepage: https://github.com/lenML/tokenizers#readme
Licenses: Apache-2.0
Latest release: 3.7.2 (published 5 months ago)
Last Synced: 2025-11-25T10:04:02.884Z (2 months ago)
Versions: 12
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1,988 Last month
Rankings:
- Dependent repos count: 30.551%
- Average: 37.143%
- Dependent packages count: 43.735%
Maintainers (1)
- luke_zhang

Dependencies

package.json npm

semver ^7.6.0

packages/baichuan2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/chatglm3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/claude/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/claude1/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/command_r_plus/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gemma/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt35turbo/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt35turbo16k/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/gpt4/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/internlm2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/llama2/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/llama3/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/qwen1_5/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_davinci002/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_davinci003/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/text_embedding_ada002/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

packages/tokenizers/package.json npm

@huggingface/jinja ^0.2.2 development
@types/node ^20.12.7 development
@xenova/transformers ^2.17.1 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development

packages/tokenizers/pnpm-lock.yaml npm

244 dependencies

packages/yi/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^

pnpm-lock.yaml npm

244 dependencies

scripts/pkg_template/package.json npm

@types/node ^20.12.7 development
ts-node ^10.9.2 development
tslib ^2.6.2 development
tsup ^8.0.2 development
typescript ^5.4.5 development
@lenml/tokenizers workspace:^