nlpo3
tiktoken-rs
| nlpo3 | tiktoken-rs | |
|---|---|---|
| 1 | 1 | |
| 30 | 200 | |
| - | - | |
| 1.6 | 7.6 | |
| 5 months ago | about 1 month ago | |
| Rust | Rust | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
nlpo3
-
Thai word tokenizers benchmark: nlpo3 vs newmm
Thanathip Suntorntip Gorlph ported Korakot Chaovavanich's Thai word tokenizer - Newmm, written in Python, to Rust called nlpo3. The nlpo3 website claimed that nlpo3 is 2X faster than Newmm. I felt that Nlpo3 must be faster than this claim because in contrast to Python's Regex engine, Rust's regex runs in the linear time since it was constrained not to support looking back/ahead. Moreover, 2X faster is ambiguous.
tiktoken-rs
What are some alternatives?
hck - A sharp cut(1) clone.
async-openai - Rust library for OpenAI
sd - Intuitive find & replace CLI (sed alternative)
opentau - Using Large Language Models for Gradual Type Inference
pythainlp - Thai Natural Language Processing in Python.
libopenai - A Rust client for OpenAI's API
regex - An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.
bytepiece-rs - The Bytepiece Tokenizer Implemented in Rust.
oso - Oso is a batteries-included framework for building authorization in your application.
rusty - AI-powered CLI tool to help you remember bash commands.
whatlang-rs - Natural language detection library for Rust. Try demo online: https://whatlang.org/
smartgpt - A program that provides LLMs with the ability to complete complex tasks using plugins.