tiktoken
JS port and JS/WASM bindings for openai/tiktoken (by dqbd)
llama-tokenizer-js
JS tokenizer for LLaMA 1 and 2 (by belladoreai)
| tiktoken | llama-tokenizer-js | |
|---|---|---|
| 1 | 5 | |
| 628 | 315 | |
| - | - | |
| 7.4 | 7.1 | |
| 28 days ago | about 2 months ago | |
| Python | JavaScript | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tiktoken
Posts with mentions or reviews of tiktoken.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
Show HN: LLaMA tokenizer that runs in browser
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
llama-tokenizer-js
Posts with mentions or reviews of llama-tokenizer-js.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
14-Jun-2023
JS tokenizer for LLaMA based LLMs (https://github.com/belladoreai/llama-tokenizer-js)
- Show HN: LLaMA tokenizer that runs in browser
- I wrote a tokenizer for LLaMA that runs inside the browser
- Understanding GPT Tokenizers
What are some alternatives?
When comparing tiktoken and llama-tokenizer-js you can also consider the following projects:
agency - Agency: Robust LLM Agent Management with Go
Constrained-Text-Generation-Studio - Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)
fastbpe - Java library implementing Byte-Pair Encoding Tokenization
Constrained-Text-Genera
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer
gpt4-tokenizer-visualizer - GPT4 Tokenizer Visualizer
gpt-tokenizer - JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.