JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
Why do you think that https://github.com/jina-ai/auto-gpt-web is a good alternative to gpt-tokenizer
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
Why do you think that https://github.com/jina-ai/auto-gpt-web is a good alternative to gpt-tokenizer