With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js. Learn more →
Top 6 TypeScript Tokenizer Projects
-
gpt-tokenizer
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
Project mention: Ohm: A library and language for building parsers, interpreters, compilers, etc. | news.ycombinator.com | 2023-10-31How does this compare with Chevrotain[1]?
More specifically, can I build lexers with Ohm? Can it generate a syntax diagram from a grammar?
[1]: https://github.com/chevrotain/chevrotain
Project mention: I wrote a tokenizer for LLaMA that runs inside the browser | /r/LocalLLaMA | 2023-06-13There are more differences between GPT2 tokenizer and LLaMA tokenizer than only the vocab and merge data. It would take me some time to do implement a GPT2 tokenizer, and there are already good alternatives for those, so it wouldn't make sense to put time into making another one. For example, this library contains a GPT2 tokenizer: https://github.com/niieani/gpt-tokenizer
Yes, this one does
https://github.com/functorism/gpt4-tokenizer-visualizer
TypeScript Tokenizer related posts
- Show HN: LLaMA tokenizer that runs in browser
- Intro video for my VS Code extension "Blockman"
- Build package for NPM & Deno
- I've seen Blockman, but is there something better (or a fix)?
- Check out my VSCode extension - Blockman - Highlight nested code blocks with boxes
-
A note from our sponsor - SurveyJS
surveyjs.io | 27 Apr 2024
Index
What are some of the best open-source Tokenizer projects in TypeScript? This list will help you:
Project | Stars | |
---|---|---|
1 | Chevrotain | 2,397 |
2 | gpt-tokenizer | 379 |
3 | vscode-blockman | 341 |
4 | gpt4-tokenizer-visualizer | 19 |
5 | leac | 5 |
6 | omni-tokenizer | 2 |
Sponsored