llama-tokenizer-js vs gpt4-tokenizer-visualizer

llama-tokenizer-js

JS tokenizer for LLaMA 1 and 2 (by belladoreai)

Suggest topics

Source Code

belladoreai.github.io

Suggest alternative

Edit details

gpt4-tokenizer-visualizer

GPT4 Tokenizer Visualizer (by functorism)

gpt4 tiktoken Tokenizer Visualization

Source Code

react-ts-h9yrjh.stackblitz.io

Suggest alternative

Edit details

SurveyJS - Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App

With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

surveyjs.io

featured

InfluxDB - Power Real-Time Data Analytics at Scale

Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

www.influxdata.com

featured

llama-tokenizer-js		gpt4-tokenizer-visualizer
	Project
5	Mentions	3
309	Stars	20
-	Growth	-
7.1	Activity	4.2
26 days ago	Latest Commit	about 1 year ago
JavaScript	Language	TypeScript
MIT License	License	-

The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

llama-tokenizer-js

Posts with mentions or reviews of llama-tokenizer-js. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-06-13.

14-Jun-2023
1 project | /r/dailyainews | 14 Jun 2023

JS tokenizer for LLaMA based LLMs (https://github.com/belladoreai/llama-tokenizer-js)
Show HN: LLaMA tokenizer that runs in browser
1 project | /r/hypeurls | 13 Jun 2023

7 projects | news.ycombinator.com | 13 Jun 2023
I wrote a tokenizer for LLaMA that runs inside the browser
2 projects | /r/LocalLLaMA | 13 Jun 2023
Understanding GPT Tokenizers
10 projects | news.ycombinator.com | 8 Jun 2023

gpt4-tokenizer-visualizer

Posts with mentions or reviews of gpt4-tokenizer-visualizer. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-06-13.

Show HN: LLaMA tokenizer that runs in browser
7 projects | news.ycombinator.com | 13 Jun 2023

Yes, this one does
https://github.com/functorism/gpt4-tokenizer-visualizer
GPT-4-Tokenizer-Visualizer
1 project | news.ycombinator.com | 21 May 2023
Gpt4-Tokenizer-Visualization
1 project | news.ycombinator.com | 21 May 2023

What are some alternatives?

When comparing llama-tokenizer-js and gpt4-tokenizer-visualizer you can also consider the following projects:

Constrained-Text-Generation-Studio - Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)

Chevrotain - Parser Building Toolkit for JavaScript

fastbpe - Java library implementing Byte-Pair Encoding Tokenization

agency - Agency: Robust LLM Agent Management with Go

Constrained-Text-Genera

vscode-blockman - VSCode extension to highlight nested code blocks

tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer

tiktoken - JS port and JS/WASM bindings for openai/tiktoken

gpt-tokenizer - JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.