awesome-llama: https://github.com/CanvaChen/chinese-llama-tokenizer

chinese llama tokenizer

Score: 2.995732273553991

Last synced: about 5 hours ago
JSON representation

Repository metadata:

目标：构建一个更符合语言学的小而美的 llama 分词器，支持中英日三国语言

• Host: GitHub
• URL: https://github.com/CanvaChen/chinese-llama-tokenizer
• Owner: CanvaChen
• License: apache-2.0
• Created: 2023-08-27T10:31:58.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2024-06-02T05:52:46.000Z (about 2 years ago)
• Last Synced: 2026-05-12T23:08:06.133Z (about 1 month ago)
• Topics: chinese, llama, tokenizer
• Language: Python
• Homepage:
• Size: 1.57 MB
• Stars: 20
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

---

Owner metadata:

• Name: Canva
• Login: CanvaChen
• Email:
• Kind: user
• Description: Software Engineer
• Website:
• Location: Fujian, China
• Twitter:
• Company:
• Icon url: https://avatars.githubusercontent.com/u/18375123?u=01f2b0961e89260e115d34bf91132602936e91bf&v=4
• Repositories: 1
• Last Synced at: 2024-07-06T05:12:19.491Z
• Profile URL: https://github.com/CanvaChen

---

GitHub Events

Total

• Watch event: 5
• Total: 5

Last Year

• Watch event: 1
• Total: 1

---

Committers metadata

Last synced: about 1 month ago

Name	Email	Commits
Canva	c****n@f****m	14

---

Issue and Pull Request metadata

Last synced: 10 months ago

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/CanvaChen/chinese-llama-tokenizer

---

---

---

Dependencies