Pure Go implementation of OpenAI's tokenizer

• tiktoken

  30 9,800 6.7 Python

  tiktoken is a fast BPE tokeniser for use with OpenAI's models.

  

I've just published a pure go version of OpenAI's tokenizer (tiktoken). If you are building applications in Go that uses any of OpenAI's API's you will quickly run into issues of requests being rejected due to inputs that exceed the max size–which is measured in tokens. This library can help you count tokens and/or split strings before you send any API requests.

• tokenizer

  2 228 4.3 Go

  Pure Go implementation of OpenAI's tiktoken tokenizer

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• tokenizer-go

  1 116 4.0 Go

  A Go wrapper for GPT-3 token encode/decode. https://platform.openai.com/tokenizer

  

Note: there is another library that does something very similar but it isn't pure go, it uses a npm package and invokes nodejs to encode/decode.

• tiktoken-go

  1 542 4.6 Go

  go version of tiktoken

It is great! I was searching for a such library several days ago. Unfortunately, I have already found another pure go port: https://github.com/pkoukk/tiktoken-go

Suggest a related project