Top 23 Tokenizer Open-Source Projects

• Chevrotain

  3 2,399 6.7 TypeScript

  Parser Building Toolkit for JavaScript

  

Project mention: Ohm: A library and language for building parsers, interpreters, compilers, etc. | news.ycombinator.com | 2023-10-31

How does this compare with Chevrotain[1]?

More specifically, can I build lexers with Ohm? Can it generate a syntax diagram from a grammar?

[1]: https://github.com/chevrotain/chevrotain

• moo

  1 807 2.4 JavaScript

  Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo. (by no-context)

  

• SurveyJS

  surveyjs.io featured

  Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

  

• kagome

  1 789 6.4 Go

  Self-contained Japanese Morphological Analyzer written in pure Go

• JFlex

  1 575 4.3 Java

  The fast scanner generator for Java™ with full Unicode support

  

• php-parser

  0 515 3.3 JavaScript

  :herb: NodeJS PHP Parser - extract AST or tokens (by glayzzle)

  

• simple

  1 489 5.6 C++

  支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin (by wangfenjin)

• js-tokens

  2 479 6.0 JavaScript

  Tiny JavaScript tokenizer.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• friso

  0 474 2.6 C

  High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

• CogCompNLP

  0 469 0.0 Java

  CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

  

• sentences

  0 421 4.5 Go

  A multilingual command line sentence tokenizer in Golang

• gpt-tokenizer

  1 380 3.3 TypeScript

  JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.

Project mention: I wrote a tokenizer for LLaMA that runs inside the browser | /r/LocalLLaMA | 2023-06-13

There are more differences between GPT2 tokenizer and LLaMA tokenizer than only the vocab and merge data. It would take me some time to do implement a GPT2 tokenizer, and there are already good alternatives for those, so it wouldn't make sense to put time into making another one. For example, this library contains a GPT2 tokenizer: https://github.com/niieani/gpt-tokenizer

• jumanpp

  1 366 1.6 C++

  Juman++ (a Morphological Analyzer Toolkit)

  

• fugashi

  1 366 5.4 C++

  A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

• lindera

  1 351 8.4 Rust

  A multilingual morphological analysis library.

  

• vscode-blockman

  8 341 3.9 TypeScript

  VSCode extension to highlight nested code blocks

• bitextor

  2 279 5.9 Python

  Bitextor generates translation memories from multilingual websites

  

• sentence-splitter

  1 203 0.0 Python

  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

  

• tiktoken-rs

  1 200 7.6 Rust

  Ready-made tokenizer library for working with GPT and tiktoken

• html5gum

  3 146 6.8 Rust

  A WHATWG-compliant HTML5 tokenizer and tag soup parser

• tokenizer

  1 140 6.1 Go

  NLP tokenizers written in Go language (by sugarme)

• simplemma

  0 125 5.2 Python

  Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

• tiktoken_ruby

  1 106 7.2 Ruby

  Unofficial ruby binding for tiktoken by way of rust

Project mention: tiktoken_ruby VS ruby-openai - a user suggested alternative | libhunt.com/r/tiktoken_ruby | 2024-05-03

• Cledev.OpenAI

  0 103 5.1 C#

  .NET 7 SDK for OpenAI with a Blazor Server playground

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Tokenizer related posts

• Show HN: LLaMA 3 tokenizer runs in the browser

  2 projects | news.ycombinator.com | 21 Apr 2024

• Show HN: LLaMA tokenizer that runs in browser

  7 projects | news.ycombinator.com | 13 Jun 2023

• Intro video for my VS Code extension "Blockman"

  3 projects | /r/vscode | 13 Jan 2023

• Build package for NPM & Deno

  5 projects | /r/Deno | 5 Jan 2023

• spaCy just got an experimental feature to detect co-references

  1 project | /r/learnmachinelearning | 7 Oct 2022

• Edit code from browser

  2 projects | /r/reactjs | 5 Jul 2022

• SpanFinder is a new experimental spaCy component that identifies span boundaries

  1 project | news.ycombinator.com | 21 Jun 2022
• A note from our sponsor - SurveyJS
  surveyjs.io | 4 May 2024

  With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com