tiktoken VS chat-with-gpt

Their tokenizer is open source: https://github.com/openai/tiktoken

Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...

OpenAI's refference tokeniser - https://github.com/openai/tiktoken

ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken

I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).

https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."

check tiktoken

you can use https://www.chatwithgpt.ai/ , I like this + it's open source.

https://github.com/cogentapps/chat-with-gpt/blob/main/app/sr...

Do you have api access? I use this repo called chat with chatgpt https://github.com/cogentapps/chat-with-gpt It literally is chatgpt website same look feel and conversation history, but you can adjust the creativeness (called temperature) the system prompt and what version you want to use.

We've been using https://github.com/cogentapps/chat-with-gpt for a while, and overall happy with it, but its development is mostly dead.

There are apps available that will take care of all this for you: https://github.com/cogentapps/chat-with-gpt is a popular one, if you need help to get it up and running just ask GPT-4 with 'browse with bing' to take a look and lead you by the hand

So far I've tried https://github.com/ztjhz/BetterChatGPT and https://github.com/cogentapps/chat-with-gpt but I'm wondering if there is anything out there more robust? A Mac desktop client would be preferred if anyone knows? I tried MacGPT but the main functionality I am looking for is an ability to sort, search and potentially tag my conversations. Any leads would be appreciated! Much thanks!