Our great sponsors
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
This might help https://github.com/google/sentencepiece
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a more popular project.
Related posts
- sentencepiece
- [P] TokenMonster Ungreedy ~ 35% faster inference and 35% increased context-length for large language models (compared to tiktoken). Benchmarks included.
- LLaMA tokenizer: is a JavaScript implementation available anywhere?
- [P] New tokenization method improves LLM performance & context-length by 25%+
- Code runs without definition of function (automatically calls a different function instead)