Our great sponsors
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
From the GitHub:
a) Comparison with SentencePiece tokenizer with comparable settings (It can also ignore word-boundaries and create phrase tokens)
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a more popular project.
Related posts
- sentencepiece
- LLaMA tokenizer: is a JavaScript implementation available anywhere?
- [P] New tokenization method improves LLM performance & context-length by 25%+
- Code runs without definition of function (automatically calls a different function instead)
- How to handle multiple languages in a sentence?