What Are Transformer Models and How Do They Work?

• tensor2tensor

  8 13,873 6.2 Python

  Discontinued Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

  

The visualisation here may be helpful.

https://github.com/tensorflow/tensor2tensor/issues/1591

• tiktoken

  30 9,800 6.7 Python

  tiktoken is a fast BPE tokeniser for use with OpenAI's models.

  

OpenAi have made their tokenizers public. [1]

As someone has pointed out, with BPE you specify the vocab size, not the token size. It's a relatively simple algo, this Huggingface course does a nice job of explaining it. [2] Plus the original paper has a very readable Python example.

[1] https://github.com/openai/tiktoken

[2] https://huggingface.co/course/chapter6/5?fw=pt

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project