tokenizer
tiktoken
tokenizer | tiktoken | |
---|---|---|
2 | 30 | |
228 | 9,800 | |
0.4% | 4.6% | |
4.3 | 6.7 | |
about 1 year ago | 19 days ago | |
Go | Python | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tokenizer
-
Understanding GPT Tokenizers
How I wish this post had appeared a few days earlier... I am writing on my own library for some agent experiments (in go, to make my life more interesting I guess), and knowing the number of tokens is important to implement a token buffer memory (as you approach the model's context window size, you prune enough messages from the beginning of the conversation that the whole thing keeps some given size, in tokens). While there's a nice native library in go for OpenAI models (https://github.com/tiktoken-go/tokenizer), the only library I found for Hugging Face models (and Claude, they published their tokenizer spec in the same JSON format) calls into HF's Rust implementation, which makes it challenging as a dependency in Go. What is more, any tokenizer needs to keep some representation of its vocabulary in memory. So, in the end I removed the true tokenizers, and ended up using an approximate version (just split it in on spaces and multiply by a factor I determined experimentally for the models I use using the real tokenizer, with a little extra for safety). If it turns out someone needs the real thing they can always provide their own token counter). I was actually rather happy with this result: I have less dependencies, and use less memory. But to get there I needed to do a deep dive too understand BPE tokenizers :)
(The library, if anyone is interested: https://github.com/ryszard/agency.)
- Pure Go implementation of OpenAI's tokenizer
tiktoken
- FLaNK AI - 01 April 2024
-
GPT-3.5 crashes when it thinks about useRalativeImagePath too much
Their tokenizer is open source: https://github.com/openai/tiktoken
Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...
-
How fast is JS tiktoken?
OpenAI's refference tokeniser - https://github.com/openai/tiktoken
-
Anthropic announces Claude 2.1 – 200k context, less refusals
ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken
-
What is the best way to get an approximate number of tokens for a piece of text?
I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).
-
Show HN: LLaMA tokenizer that runs in browser
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
-
Made a GPT-3.5-Turbo and GPT-4 Tokenizer
It's built on top of the tiktoken library and is basically just a lambda function in the backend.
- AiPrice - an API for calculating OpenAI tokens and pricing
-
Anyone able to explain what happened here?
"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."
-
Which lib is the tokenizer page using to calculate the tokens?
check tiktoken
What are some alternatives?
llama.go - llama.go is like llama.cpp in pure Golang!
daath-ai-parser - Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.
agency - Agency: Robust LLM Agent Management with Go
CLIP - CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
sentences - A multilingual command line sentence tokenizer in Golang
skypilot - SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.
Constrained-Text-Genera
bricks - Open-source natural language enrichments at your fingertips.
tokenizer-go - A Go wrapper for GPT-3 token encode/decode. https://platform.openai.com/tokenizer
terminal-copilot - A smart terminal assistant that helps you find the right command.
Constrained-Text-Generation-Studio - Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)
jupyter-scheduler - Run Jupyter notebooks as jobs