Top 7 Rust Natural Language Processing Projects

tokenizers

8 8,395 8.5 Rust

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Project mention: HF Transfer: Speed up file transfers | /r/rust | 2023-07-07

Hugging Face seems to like Rust. They also wrote Tokenizers in Rust.

lingua-rs

9 817 8.9 Rust

The most accurate natural language detection library for Rust, suitable for short text and mixed-language text

Project mention: I created a program that finds out which anki cards out of 50_000 are in english and deletes them in 2 minutes | /r/rust | 2023-10-23

Discovery of Lingua: While working on a different project, I discovered the Lingua library.

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
nlprule

5 570 0.0 Rust

A fast, low-resource Natural Language Processing and Text Correction library written in Rust.

Project mention: Vale.sh – A Linter for Prose | news.ycombinator.com | 2023-09-03

Another interesting projects in the space:
- nlprule: https://github.com/bminixhofer/nlprule
- prosemd: https://github.com/kitten/prosemd-lsp
- cargo spellcheck: https://github.com/drahnr/cargo-spellcheck

whichlang

3 341 5.1 Rust

A blazingly fast and lightweight language detection library for Rust

Project mention: Show HN: Quickwit – Cost-efficient Elasticsearch alternative on object storage | news.ycombinator.com | 2023-06-07

- Another nice comment seen on HN « it seems to be very easy to run, not very IO intensive, and running fine on a single node with modest hardware with >2 billion log rows. It has a really cool dynamic schema feature too.» [9]
Fun fact: at least 4 users are using Garage[10] as the object storage, this OSS project looks really promising and made the HN front page a few months ago[11], we really cherish the OSS for this kind of unexpected combination.
Any feedback positive/negative always greatly appreciated here!
[0] Quickwit repo: https://github.com/quickwit-oss/quickwit
[1] Searching the web under 1000$/month: https://news.ycombinator.com/item?id=27074481
[2] Chitchat gossip library: https://github.com/quickwit-oss/chitchat
[3] Columnar format: https://github.com/quickwit-oss/tantivy/tree/main/columnar
[4] Tantivy library: https://github.com/quickwit-oss/tantivy/
[5] Whichlang library: https://github.com/quickwit-oss/whichlang
[6] GitHub Archive demo in terminal: https://www.youtube.com/watch?v=SNq3bARRlDI
[7] Indexing performance: https://twitter.com/fulmicoton/status/1638016949459488768
[8] https://twitter.com/arnonrgo/status/1645429632303235073?s=20
[9] https://news.ycombinator.com/item?id=35742544
[10] Garage object storage: https://garagehq.deuxfleurs.fr/
[11] https://news.ycombinator.com/item?id=33853539

instant-segment

1 85 7.0 Rust

Fast English word segmentation in Rust
dpar

0 41 3.2 Rust

Neural network transition-based dependency parser (in Rust)
nlpo3

1 30 1.6 Rust

Thai Natural Language Processing library in Rust, with Python and Node bindings.
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).