llama-tokenizer-js
JS tokenizer for LLaMA 1 and 2 (by belladoreai)
tiktoken
JS port and JS/WASM bindings for openai/tiktoken (by dqbd)
SurveyJS - Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
surveyjs.io
featured
| llama-tokenizer-js | tiktoken | |
|---|---|---|
| 5 | 1 | |
| 309 | 617 | |
| - | - | |
| 7.1 | 7.4 | |
| 26 days ago | 8 days ago | |
| JavaScript | Python | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
llama-tokenizer-js
Posts with mentions or reviews of llama-tokenizer-js.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
14-Jun-2023
JS tokenizer for LLaMA based LLMs (https://github.com/belladoreai/llama-tokenizer-js)
- Show HN: LLaMA tokenizer that runs in browser
- I wrote a tokenizer for LLaMA that runs inside the browser
- Understanding GPT Tokenizers
tiktoken
Posts with mentions or reviews of tiktoken.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-06-13.
-
Show HN: LLaMA tokenizer that runs in browser
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
What are some alternatives?
When comparing llama-tokenizer-js and tiktoken you can also consider the following projects:
Constrained-Text-Generation-Studio - Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)
agency - Agency: Robust LLM Agent Management with Go
fastbpe - Java library implementing Byte-Pair Encoding Tokenization
Constrained-Text-Genera
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer
gpt4-tokenizer-visualizer - GPT4 Tokenizer Visualizer
gpt-tokenizer - JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.