tokenizers
BlingFire
| tokenizers | BlingFire | |
|---|---|---|
| 8 | 2 | |
| 8,424 | 1,781 | |
| 1.6% | 0.3% | |
| 8.5 | 3.6 | |
| 4 days ago | 6 months ago | |
| Rust | C++ | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tokenizers
-
HF Transfer: Speed up file transfers
Hugging Face seems to like Rust. They also wrote Tokenizers in Rust.
-
LLM custom dictionary
Your intuition is right. There are two ways (in increasing order of result performance) : 1. You can simply extend vocab file of the tokenizer and test the predictions 2. You can extend the vocab file and re-train your model on custom data which has these new tokens. Check the following issue on GitHub : https://github.com/huggingface/tokenizers/issues/247
-
[D] SentencePiece, WordPiece, BPE... Which tokenizer is the best one?
SentencePiece -> implementation of some algorithms (there are several others, https://github.com/microsoft/BlingFire https://github.com/glample/fastBPE https://github.com/huggingface/tokenizers )
-
Portability of Rust in 2021
In sum I would like the idea to go with Rust as I more or less got to rewrite the whole thing anyway, but I am a bit skeptical if I will be able to interface with everything that might come up at some point. Or probably end up in a wrapper hell if I got to use more C++ libraries. On the other hand there are definitely a few Rust projects out there that might come in handy (for example https://github.com/huggingface/tokenizers). And the build process is pretty awful right now (CMake it is but with lots of hacks).
-
[D] What's going to be the dominant language for machine learning in 5 years?
A full machine learning pipeline usually comprises far more than just the model, and this is the area where Rust may shine (the recent work by HuggingFace and their https://github.com/huggingface/tokenizers library is a good example)
-
substitute for tokenizer in torchtext
As for other tokenizers, you can take a look at - Huggingface tokenizers library: https://github.com/huggingface/tokenizers - NLTK tokenize: https://www.nltk.org/api/nltk.tokenize.html - Polygot: https://pypi.org/project/polyglot/
-
PyO3: Rust Bindings for the Python Interpreter
Huggingface Tokenizers (https://github.com/huggingface/tokenizers), which are now used by default in their Transformers Python library, use pyO3 and became popular due to the pitch that it encoded text an order of magnitude faster with zero config changes.
It lives up to that claim. (I had issues with return object typing when going between Python/Rust at first but those are more consistent now)
-
Rusticles #19 - Wed Nov 11 2020
huggingface/tokenizers (Rust): 💥Fast State-of-the-Art Tokenizers optimized for Research and Production
BlingFire
-
[D] SentencePiece, WordPiece, BPE... Which tokenizer is the best one?
SentencePiece -> implementation of some algorithms (there are several others, https://github.com/microsoft/BlingFire https://github.com/glample/fastBPE https://github.com/huggingface/tokenizers )
-
Ask HN: Who is hiring? (March 2021)
• Develop the best technology to bring deep learning solutions to unprecedented scale, for example we built the world's fastest tokenizer. [https://github.com/microsoft/BlingFire]
What are some alternatives?
onnx-tensorflow - Tensorflow Backend for ONNX
Mattermost - Mattermost is an open source platform for secure collaboration across the entire software development lifecycle..
onnxruntime - ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
OpenKP - Automatically extracting keyphrases that are salient to the document meanings is an essential step to semantic document understanding. An effective keyphrase extraction (KPE) system can benefit a wide range of natural language processing and information retrieval tasks. Recent neural methods formulate the task as a document-to-keyphrase sequence-to-sequence task. These seq2seq learning models have shown promising results compared to previous KPE systems The recent progress in neural KPE is mostly observed in documents originating from the scientific domain. In real-world scenarios, most potential applications of KPE deal with diverse documents originating from sparse sources. These documents are unlikely to include the structure, prose and be as well written as scientific papers. They often include a much diverse document structure and reside in various domains whose contents target much wider audiences than scientists. To encourage the research community to develop a powerful neural m
setuptools-rust - Setuptools plugin for Rust support
sgr - sgr (command line client for Splitgraph) and the splitgraph Python library
rayon - Rayon: A data parallelism library for Rust
python-fake-data-producer-for-apache-kafka - The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and push it to an Apache Kafka topic.
tch-rs - Rust bindings for the C++ api of PyTorch.
fargate-game-servers - This repository contains an example solution on how to scale a fleet of game servers on AWS Fargate on Elastic Container Service and route players to game sessions using a Serverless backend. Game Server data is stored in ElastiCache Redis. All resources are deployed with Infrastructure as Code using CloudFormation, Serverless Application Model, Docker and bash/powershell scripts. By leveraging AWS Fargate for your game servers you don't need to manage the underlying virtual machines.
rust-bert - Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)
parabol - Free online agile retrospective meeting tool