tokenizer-go
A Go wrapper for GPT-3 token encode/decode. https://platform.openai.com/tokenizer (by pandodao)
tiktoken-go
go version of tiktoken (by pkoukk)
| tokenizer-go | tiktoken-go | |
|---|---|---|
| 1 | 1 | |
| 116 | 560 | |
| 0.9% | - | |
| 4.0 | 3.7 | |
| about 1 year ago | 6 days ago | |
| Go | Go | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tokenizer-go
Posts with mentions or reviews of tokenizer-go.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-04-07.
-
Pure Go implementation of OpenAI's tokenizer
Note: there is another library that does something very similar but it isn't pure go, it uses a npm package and invokes nodejs to encode/decode.
tiktoken-go
Posts with mentions or reviews of tiktoken-go.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-04-07.
-
Pure Go implementation of OpenAI's tokenizer
It is great! I was searching for a such library several days ago. Unfortunately, I have already found another pure go port: https://github.com/pkoukk/tiktoken-go
What are some alternatives?
When comparing tokenizer-go and tiktoken-go you can also consider the following projects:
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer
jotbot - JotBot generates the missing code documentation for your Go and TypeScript projects. Powered by AI.
chatgptui - ChatGPT 🤖 with Textual User Interface (TUI) mode written in Go.
tiktoken - tiktoken is a fast BPE tokeniser for use with OpenAI's models.
go-openai - OpenAI ChatGPT, GPT-3, GPT-4, DALL·E, Whisper API wrapper for Go