tiktoken
JS port and JS/WASM bindings for openai/tiktoken (by dqbd)
gpt4-tokenizer-visualizer
GPT4 Tokenizer Visualizer (by functorism)
tiktoken | gpt4-tokenizer-visualizer | |
---|---|---|
1 | 3 | |
633 | 20 | |
- | - | |
7.4 | 4.2 | |
about 1 month ago | about 1 year ago | |
Python | TypeScript | |
MIT License | - |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tiktoken
Posts with mentions or reviews of tiktoken.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
Show HN: LLaMA tokenizer that runs in browser
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
gpt4-tokenizer-visualizer
Posts with mentions or reviews of gpt4-tokenizer-visualizer.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
Show HN: LLaMA tokenizer that runs in browser
Yes, this one does
https://github.com/functorism/gpt4-tokenizer-visualizer
- GPT-4-Tokenizer-Visualizer
- Gpt4-Tokenizer-Visualization
What are some alternatives?
When comparing tiktoken and gpt4-tokenizer-visualizer you can also consider the following projects:
llama-tokenizer-js - JS tokenizer for LLaMA 1 and 2
Chevrotain - Parser Building Toolkit for JavaScript
agency - Agency: Robust LLM Agent Management with Go
vscode-blockman - VSCode extension to highlight nested code blocks