Open-source projects categorized as Tokenization | Edit details
Language filter: + Python + C + Go + HTML

Top 6 Tokenization Open-Source Projects

  • GitHub repo Ravencoin

    Ravencoin Core integration/staging tree

    Project mention: What is Ravencoin? (alert: this is not a hype-post) | reddit.com/r/Ravencoin | 2021-05-07

    Most of the recent commits are for the ravencoin.org website while latest commit to ravencoin repository is 17-Jan. Some works for RVN core is being done on PRs but half (3/6) are about GUI & document. Also, roadmap doesn't have ETA (https://github.com/RavenProject/Ravencoin/blob/master/roadmap/README.md).

  • GitHub repo Databunker

    Secure storage for personal records built to comply with GDPR

    Project mention: Hottest Israel Startup with Open-Source Spirit | reddit.com/r/Israel | 2021-04-08

    Check out the project website for additional information: https://databunker.org/

  • GitHub repo trankit

    Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

    Project mention: Trankit v1.0.0 - An open-source Transformer-based Multilingual NLP Toolkit for 56 languages is out. | reddit.com/r/LanguageTechnology | 2021-03-31

    Trankit is written in Python and can be easily installed via pip. Our code and pretrained models are publicly available at: https://github.com/nlp-uoregon/trankit

  • GitHub repo TokenScript

    TokenScript schema, specs and paper

    Project mention: Daily General Discussion - March 9, 2021 | reddit.com/r/ethfinance | 2021-03-09

    AlphaWallet introduces TokenScript to create SmartTokens, traditional ERC20 or ERC721 tokens with extendable structure and signed JavaScript.

  • GitHub repo python-fpe

    FPE - Format Preserving Encryption with FF3 in Python

    Project mention: Release of format preserving encryption in Python | news.ycombinator.com | 2021-03-15
  • GitHub repo wink-eng-lite-model

    English lite language model for wink-nlp.

    Project mention: How to tokenize a string? | dev.to | 2021-02-09

    To tokenize a string using winkNLP, read the text using readDoc. Then use the tokens method to extract a collection of tokens from the string. Follow this with the out method to get this collection as a JavaScript array. This is how you can tokenize a string:

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-05-07.


What are some of the best open-source Tokenization projects? This list will help you:

Project Stars
1 Ravencoin 798
2 Databunker 747
3 trankit 446
4 TokenScript 95
5 python-fpe 13
6 wink-eng-lite-model 4