tiktoken VS devspace

Their tokenizer is open source: https://github.com/openai/tiktoken

Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...

OpenAI's refference tokeniser - https://github.com/openai/tiktoken

ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken

I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).

https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."

check tiktoken

Check also devspace.sh and okteto.com

There is actually a built-in converter from docker-compose.yaml to devspace.yaml, it is part of the "devspace init" command. There are a few known issues related to it, in case you would like to contribute :)

Wrapping https://devspace.sh in my company's internal CLI to improve development experience

For Kubernetes clusters, https://devspace.sh/ allows you to iterate on your app in a running pod from your IDE. No ci/cd pipeline, no re-applying manifests; it just uploads source code from your local machine and restarts your app for you in an existing pod. Pretty slick!!! But it requires K8s, so some devs may balk at it. (Tilt and Skaffold do the same thing)

We’re excited to announce that DevSpace version 6 has been released. Thanks to everyone who used the alpha and beta versions and gave us feedback.

DevSpace is free, lets you develop natively in the cloud on any K8s cluster. Compatible with any multitenancy solution, policy governance, etc. Works with your IDE of choice. Declarative configuration. Supports custom pipelines and deploy logic. Can link multiple environments together. https://devspace.sh/

My favourite tool in this category is Devspace. As you've requested it is capable of building and deploying multiple images.