Top 4 Rust Bert Projects

tokenizers

8 8,395 8.5 Rust

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Project mention: HF Transfer: Speed up file transfers | /r/rust | 2023-07-07

Hugging Face seems to like Rust. They also wrote Tokenizers in Rust.

rust-bert

7 2,415 6.8 Rust

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

Project mention: How to leverage the state-of-the-art NLP models in Rust | /r/infinilabs | 2023-06-07

brew install libtorch brew link libtorch brew ls --verbose libtorch | grep dylib export LIBTORCH=$(brew --cellar pytorch)/$(brew info --json pytorch | jq -r '.[0].installed[0].version') export LD_LIBRARY_PATH=${LIBTORCH}/lib:$LD_LIBRARY_PATH git clone https://github.com/guillaume-be/rust-bert.git cd rust-bert ORT_STRATEGY=system cargo run --example sentence_embeddings

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
syntaxdot

4 65 6.2 Rust

Neural syntax annotator, supporting sequence labeling, lemmatization, and dependency parsing.

Project mention: Candle: Torch Replacement in Rust | news.ycombinator.com | 2023-08-08

I am so happy about them releasing this. A few years ago I wrote a multi-task syntax annotator in Rust using Laurent Mazare's excellent tch-rs binding (it seems like he is also working on Candle):
https://github.com/tensordot/syntaxdot
However, the deployment story was always quite difficult. The PyTorch C++ API is not stable, so a particular version of tch-rs will only work with a particular PyTorch version. So, anyone wanting to use SyntaxDot always had to get exactly the right version of libtorch (and set some environment variables) to build the project.
The idea of making an abstraction over Torch and Rust ndarray (similar to Burn) crossed my mind several times, but there is only so much that I could do as a solo developer. So Candle would be a god-given if I was still working on this project.
Seeing Candle wants to make me port curated-transformers to Candle for fun:
https://github.com/explosion/curated-transformers

semdesk

1 1 4.5 Rust

Semantic Desktop Search - search for answers not the file names

Project mention: hobby attempt at creating a local desktop semantic search | /r/LanguageTechnology | 2023-04-26

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).