tiktoken VS min

Their tokenizer is open source: https://github.com/openai/tiktoken

Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...

OpenAI's refference tokeniser - https://github.com/openai/tiktoken

ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken

I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).

https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."

check tiktoken

FWIW, I found https://minbrowser.org/ and it uses very little CPU/Memory, but it uses DuckDuckGo and I prefer to use Google as my search engine... Also I do not think it supports installation of extensions?...

I tried Min, I found it both appealing and confusing at the same time. It has a minimalist design and it simply looks amazing, but it has a massive drawback: It is TOO MUCH minimalist, you get confused using it, also because it's very unintuitive to use at first. It's also one of the slowest I've ever seen (on Windows, idk about Mac), even without extensions (because yeah, even if it's Chromium-based, you can't download any extensions). I guess if dev's increase it's performances, add extension's compatibility, and make it less confusing, it would be a pretty good browser, but for now, it sleeps in my trashcan.

In the CachyOS installation itself you can select to install Firefox. I hope they leave that firefox fork behind and do something super-vitaminized with Min Browser. I think they could improve a lot making custom userscripts

Some browsers (such as Min Browser) don't have a status bar like Firefox or Chrome to preview the target (it can only be displayed in the context menu).

I know it is not but the browser reminded me https://minbrowser.org/

You can use Min, Fluid or any browser with full screen mode to have the same effect.

I love firefox, but the UI is to old, I like the Min Browser look

des navigateurs il y en a plein: min, puffin, otter, falkon, dot, privacy browser, orion, netsurf, maxthon (de mémoire il est douteux en termes de vie privée celui là), etc.