tokenizer
NLP tokenizers written in Go language (by sugarme)
maleeni
A lexer generator for golang (by nihei9)
| tokenizer | maleeni | |
|---|---|---|
| 1 | 1 | |
| 140 | 5 | |
| - | - | |
| 6.1 | 0.0 | |
| 2 months ago | about 2 years ago | |
| Go | Go | |
| Apache License 2.0 | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tokenizer
Posts with mentions or reviews of tokenizer.
We have used some of these posts to build our list of alternatives
and similar projects.
-
Golang Modules Tutorial
The github repository of tokenizer can be checked here.
maleeni
Posts with mentions or reviews of maleeni.
We have used some of these posts to build our list of alternatives
and similar projects.
-
maleeni: yet another lexer generator for golang
I have implemented a lexer generator for golang. The repository is here github.com/nihei9/maleeni.
What are some alternatives?
When comparing tokenizer and maleeni you can also consider the following projects:
spaGO - Self-contained Machine Learning and Natural Language Processing library in Go
libfsm - DFA regular expression library & friends
sentences - A multilingual command line sentence tokenizer in Golang
netz - Discover internet-wide misconfigurations while drinking coffee
kagome - Self-contained Japanese Morphological Analyzer written in pure Go
tokenizer - Tokenizer (lexer) for golang
participle - A parser library for Go
chris - Pratt parser implementation in Go
goselect - SQL like 'select' interface for files
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer