[D] How do pretrained tokenizers work?

• transformers

  175 124,557 10.0 Python

  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

  

I have been using the pretrained tokenizers available from the huggingface/transformers library. And they have been working well for my use case.

• sentencepiece

  19 9,451 8.3 C++

  Unsupervised text tokenizer for Neural Network-based text generation.

  

For papers, take a look at references here https://github.com/google/sentencepiece

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project