awesome-llama: https://github.com/ReshiAdavan/Thoth

bytepairencoding gpt-2 gpt-4 llama2 natural-language-processing python rust sentencepiece tiktoken tokenizer

Score: -Infinity

Last synced: about 4 hours ago
JSON representation

Repository metadata:

tokenizer for large-scale language models (GPT, Claude, Llama, etc.)

• Host: GitHub
• URL: https://github.com/ReshiAdavan/Thoth
• Owner: ReshiAdavan
• Created: 2024-03-01T21:02:20.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2024-06-29T15:44:35.000Z (almost 2 years ago)
• Last Synced: 2026-06-09T12:03:50.139Z (10 days ago)
• Topics: bytepairencoding, gpt-2, gpt-4, llama2, natural-language-processing, python, rust, sentencepiece, tiktoken, tokenizer
• Language: Python
• Homepage:
• Size: 2.96 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

---

Owner metadata:

• Name: Reshi Adavan
• Login: ReshiAdavan
• Email:
• Kind: user
• Description: Software Dev @ Achievers | UWaterloo CE
• Website: https://www.linkedin.com/in/reshiadavan/
• Location: Markham, Ontario
• Twitter:
• Company:
• Icon url: https://avatars.githubusercontent.com/u/70111614?u=3f017d77d6196c1adb8a3ca9a6550badf5753c00&v=4
• Repositories: 7
• Last Synced at: 2023-03-19T17:39:27.075Z
• Profile URL: https://github.com/ReshiAdavan

---

GitHub Events

Total

• Total: 0

Last Year

• Total: 0

---

Committers metadata

Last synced: 8 days ago

Name	Email	Commits
ReshiAdavan	r****7@g****m	40

---

Issue and Pull Request metadata

Last synced: about 2 months ago

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/ReshiAdavan/Thoth

---

---

---

Dependencies