tiktoken VS reflex

Their tokenizer is open source: https://github.com/openai/tiktoken

Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...

OpenAI's refference tokeniser - https://github.com/openai/tiktoken

ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken

I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).

https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."

check tiktoken

Hey thanks for the feedback. We're working on relaxing our dependencies [1] to make reflex more compatible. Do you remember what libraries you had the conflict with?

[1] https://github.com/reflex-dev/reflex/pull/2796

Thanks! Pue looks cool, thanks for sharing. I see some similarities to https://reflex.dev in terms of providing a declarative dom expression language with built-in conditionals and loop primitives.

In the dynamic world of web development, Python has emerged as a dominant force, especially in backend development – the primary focus of this blog post. Although it's worth mentioning that there are ongoing efforts to use Python for the frontend as well, like Reflex (previously known as Pynecone, they presumably had to change their name because of Pinecone vector database), which even garnered support from Y Combinator. Samuel Colvin (creator of Pydantic) is also working on FastUI (he literally just released the first version in December 2023).

They have a ready to use LLM chat App, which makes it more likely I will check it out.

https://github.com/reflex-dev/reflex

We will use Reflex to build this chatbot.

To learn more about Reflex, you can read here: https://reflex.dev/

Reflex is an open-source, full-stack Python framework that makes it easy to build and deploy web apps in minutes. You have most of the features of a frontend library like Reactjs and a backend framework like Django in one with ease in development and deployment. All while developing in a single language PYTHON.

Repo : https://github.com/reflex-dev/reflex