Gpt-tokenizer Alternatives
Similar projects and alternatives to gpt-tokenizer based on common topics and language
-
Lobe Chat
LobeChat is a open-source, extensible (Function Calling), high-performance chatbot framework.It supports one-click free deployment of your private ChatGPT/LLM web application.
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
-
bpe-encoder-php
BPE (Byte-Pair Encoding) Encoder Decoder for OpenAI's GPT-2 / GPT-3 Implemented In Pure PHP, Zero Dependency, Multi Byte Supported.
-
XENA
XENA is the managed remote administration platform for botnet creation & development powered by blockchain and machine learning. Aiming to provide an ecosystem which serves the bot herders. Favoring secrecy and resiliency over performance. It's micro-service oriented allowing for specialization and lower footprint. Join the community of the ultimate Botnet kit.
gpt-tokenizer reviews and mentions
-
I wrote a tokenizer for LLaMA that runs inside the browser
There are more differences between GPT2 tokenizer and LLaMA tokenizer than only the vocab and merge data. It would take me some time to do implement a GPT2 tokenizer, and there are already good alternatives for those, so it wouldn't make sense to put time into making another one. For example, this library contains a GPT2 tokenizer: https://github.com/niieani/gpt-tokenizer
Stats
niieani/gpt-tokenizer is an open source project licensed under MIT License which is an OSI approved license.
The primary programming language of gpt-tokenizer is TypeScript.
Popular Comparisons
Sponsored