fastbpe
Java library implementing Byte-Pair Encoding Tokenization (by deepanprabhu)
llama-tokenizer-js
JS tokenizer for LLaMA and LLaMA 2 (by belladoreai)
| fastbpe | llama-tokenizer-js | |
|---|---|---|
| 1 | 5 | |
| 2 | 305 | |
| - | - | |
| 5.5 | 7.1 | |
| 12 months ago | 20 days ago | |
| Java | JavaScript | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fastbpe
Posts with mentions or reviews of fastbpe.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-08.
-
Understanding GPT Tokenizers
Tokenization is very important and I did implement fastbpe in java to understand things - https://github.com/deepanprabhu/fastbpe
llama-tokenizer-js
Posts with mentions or reviews of llama-tokenizer-js.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
14-Jun-2023
JS tokenizer for LLaMA based LLMs (https://github.com/belladoreai/llama-tokenizer-js)
- Show HN: LLaMA tokenizer that runs in browser
- I wrote a tokenizer for LLaMA that runs inside the browser
- Understanding GPT Tokenizers
What are some alternatives?
When comparing fastbpe and llama-tokenizer-js you can also consider the following projects:
Constrained-Text-Genera
Constrained-Text-Generation-Studio - Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)
llama.go - llama.go is like llama.cpp in pure Golang!
agency - Agency: Robust LLM Agent Management with Go
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer
tiktoken - JS port and JS/WASM bindings for openai/tiktoken
gpt4-tokenizer-visualizer - GPT4 Tokenizer Visualizer
gpt-tokenizer - JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
fastbpe vs Constrained-Text-Genera
llama-tokenizer-js vs Constrained-Text-Generation-Studio
fastbpe vs Constrained-Text-Generation-Studio
llama-tokenizer-js vs Constrained-Text-Genera
fastbpe vs llama.go
llama-tokenizer-js vs agency
fastbpe vs agency
llama-tokenizer-js vs tokenizer
llama-tokenizer-js vs tiktoken
llama-tokenizer-js vs gpt4-tokenizer-visualizer
llama-tokenizer-js vs gpt-tokenizer