tokenizer
llama.go
tokenizer | llama.go | |
---|---|---|
2 | 12 | |
228 | 1,160 | |
0.4% | - | |
4.3 | 8.2 | |
about 1 year ago | 5 months ago | |
Go | Go | |
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tokenizer
-
Understanding GPT Tokenizers
How I wish this post had appeared a few days earlier... I am writing on my own library for some agent experiments (in go, to make my life more interesting I guess), and knowing the number of tokens is important to implement a token buffer memory (as you approach the model's context window size, you prune enough messages from the beginning of the conversation that the whole thing keeps some given size, in tokens). While there's a nice native library in go for OpenAI models (https://github.com/tiktoken-go/tokenizer), the only library I found for Hugging Face models (and Claude, they published their tokenizer spec in the same JSON format) calls into HF's Rust implementation, which makes it challenging as a dependency in Go. What is more, any tokenizer needs to keep some representation of its vocabulary in memory. So, in the end I removed the true tokenizers, and ended up using an approximate version (just split it in on spaces and multiply by a factor I determined experimentally for the models I use using the real tokenizer, with a little extra for safety). If it turns out someone needs the real thing they can always provide their own token counter). I was actually rather happy with this result: I have less dependencies, and use less memory. But to get there I needed to do a deep dive too understand BPE tokenizers :)
(The library, if anyone is interested: https://github.com/ryszard/agency.)
- Pure Go implementation of OpenAI's tokenizer
llama.go
-
Understanding GPT Tokenizers
You might reuse simple LLaMA tokenizer right in your Go code, look there:
https://github.com/gotzmann/llama.go/blob/8cc54ca81e6bfbce25...
-
April 2023
llama.go is like llama.cpp in pure Golang (https://github.com/gotzmann/llama.go)
- llama.go v1.4 - introduces Rest API for your GPT services
- [Golang] Llama.go - Meta's Llama GPT Inférence dans Pure Golang
- LLaMA.go v1.4: now with scalable REST API exposing local GPT model
- Local LLaMA REST API with llama.go v1.4
- LLaMA.go v1.4 - introducing REST API for building your own GPT services
-
MiniGPT-4
I'm developing framework [1] in Golang with this goal in mind :) It successfully runs relatively big LLM right now, and diffusion models will be the next step
[1] https://github.com/gotzmann/llama.go/
- gotzmann/llama.go: llama.go is like llama.cpp in pure Golang!
- Show HN: Llama.go – port of llama.cpp to pure Go
What are some alternatives?
tiktoken - tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Flowise - Drag & drop UI to build your customized LLM flow
agency - Agency: Robust LLM Agent Management with Go
gpt4all.unity - Bindings of gpt4all language models for Unity3d running on your local machine
sentences - A multilingual command line sentence tokenizer in Golang
nn-zero-to-hero - Neural Networks: Zero to Hero
Constrained-Text-Genera
LLamaStack - ASP.NET Core Web, WebApi & WPF implementations for LLama.cpp & LLamaSharp
tokenizer-go - A Go wrapper for GPT-3 token encode/decode. https://platform.openai.com/tokenizer
langchain-alpaca - Run Alpaca LLM in LangChain
Constrained-Text-Generation-Studio - Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)
prompt-engineering - ChatGPT Prompt Engineering for Developers - deeplearning.ai