Top 7 Rust Natural Language Processing Projects

tokenizers

2 7,583 0.0 Rust

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Project mention: HF Transfer: Speed up file transfers | /r/rust | 2023-07-07

Hugging Face seems to like Rust. They also wrote Tokenizers in Rust.
lingua-rs

2 724 0.0 Rust

The most accurate natural language detection library for Rust, suitable for long and short text alike

Project mention: Lingua 1.5.0 - The most accurate natural language detection library for Rust, now with support for detecting multiple languages in mixed-language text | /r/rust | 2023-06-15
Mergify

blog.mergify.com
sponsored

Updating dependencies is time-consuming.. Solutions like Dependabot or Renovate update but don't merge dependencies. You need to do it manually while it could be fully automated! Add a Merge Queue to your workflow and stop caring about PR management & merging. Try Mergify for free.
nlprule

1 541 0.0 Rust

A fast, low-resource Natural Language Processing and Text Correction library written in Rust.

Project mention: Vale.sh – A Linter for Prose | news.ycombinator.com | 2023-09-03

Another interesting projects in the space:
- nlprule: https://github.com/bminixhofer/nlprule
- prosemd: https://github.com/kitten/prosemd-lsp
- cargo spellcheck: https://github.com/drahnr/cargo-spellcheck
whichlang

3 322 10.0 Rust

A blazingly fast and lightweight language detection library for Rust

Project mention: Show HN: Quickwit – Cost-efficient Elasticsearch alternative on object storage | news.ycombinator.com | 2023-06-07

- Another nice comment seen on HN « it seems to be very easy to run, not very IO intensive, and running fine on a single node with modest hardware with >2 billion log rows. It has a really cool dynamic schema feature too.» [9]
Fun fact: at least 4 users are using Garage[10] as the object storage, this OSS project looks really promising and made the HN front page a few months ago[11], we really cherish the OSS for this kind of unexpected combination.
Any feedback positive/negative always greatly appreciated here!
[0] Quickwit repo: https://github.com/quickwit-oss/quickwit
[1] Searching the web under 1000$/month: https://news.ycombinator.com/item?id=27074481
[2] Chitchat gossip library: https://github.com/quickwit-oss/chitchat
[3] Columnar format: https://github.com/quickwit-oss/tantivy/tree/main/columnar
[4] Tantivy library: https://github.com/quickwit-oss/tantivy/
[5] Whichlang library: https://github.com/quickwit-oss/whichlang
[6] GitHub Archive demo in terminal: https://www.youtube.com/watch?v=SNq3bARRlDI
[7] Indexing performance: https://twitter.com/fulmicoton/status/1638016949459488768
[8] https://twitter.com/arnonrgo/status/1645429632303235073?s=20
[9] https://news.ycombinator.com/item?id=35742544
[10] Garage object storage: https://garagehq.deuxfleurs.fr/
[11] https://news.ycombinator.com/item?id=33853539
instant-segment

0 75 0.0 Rust

Fast English word segmentation in Rust
dpar

0 41 3.2 Rust

Neural network transition-based dependency parser (in Rust)
nlpo3

0 27 0.0 Rust

Thai Natural Language Processing library in Rust, with Python and Node bindings.
SonarQube

www.sonarqube.org
sponsored

Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-09-03.