[D] SentencePiece, WordPiece, BPE... Which tokenizer is the best one?

• BlingFire

  2 1,776 3.6 C++

  A lightning fast Finite State machine and REgular expression manipulation library.

  

SentencePiece -> implementation of some algorithms (there are several others, https://github.com/microsoft/BlingFire https://github.com/glample/fastBPE https://github.com/huggingface/tokenizers )

• fastBPE

  1 641 0.0 C++

  Fast BPE

SentencePiece -> implementation of some algorithms (there are several others, https://github.com/microsoft/BlingFire https://github.com/glample/fastBPE https://github.com/huggingface/tokenizers )

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• tokenizers

  8 8,375 8.5 Rust

  💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

  

SentencePiece -> implementation of some algorithms (there are several others, https://github.com/microsoft/BlingFire https://github.com/glample/fastBPE https://github.com/huggingface/tokenizers )

Suggest a related project