tiktoken VS openai-python

Their tokenizer is open source: https://github.com/openai/tiktoken

Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...

OpenAI's refference tokeniser - https://github.com/openai/tiktoken

ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken

I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).

https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."

check tiktoken

The ongoing model anchoring/grounding issue likely affects all GPT-4 checkpoints/variants, but is most prominent with the latest "gpt-4-turbo-2024-04-09" variant due to its most recent cutoff date, might imply deeper issues with the current model architecture, or at least how it's been updated:

https://github.com/openai/openai-python/issues/1310

See also the original thread on OpenAI's developer forums (linked on the GitHub issue) with confirmations from others.

A test code snippet is included in the GitHub issue to A/B test the problem yourself with your own questions if need be.

- Iterate on large codebases quickly: we use Rust for maximum performance

GritQL has already been used on thousands of repositories for complex migrations[1] but we're excited to collaborate more with the open source community.

[1] Ex. https://github.com/openai/openai-python/discussions/742

Explore OpenAI and learn how to easily integrate it into your existing application with the OpenAI Python module.

You tried to access openai.Completion, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API. You can run `openai migrate` to automatically upgrade your codebase to use the 1.0.0 interface. Alternatively, you can pin your installation to the old version, e.g. `pip install openai==0.28` A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742

Relatedly, I checked and OpenAI deleted all references to their ChatML spec from their GitHub repo.

This is what it said in an earlier commit: https://github.com/openai/openai-python/blob/2942bf4bb635b1e...

Make sure to install the OpenAI library first. Since I'm using Python, I need

Source: https://github.com/openai/openai-python/discussions/742