Tokenizer-go Alternatives

Similar projects and alternatives to tokenizer-go

• tiktoken

  30 9,800 6.7 Python tokenizer-go VS tiktoken

  tiktoken is a fast BPE tokeniser for use with OpenAI's models.

  

• tokenizer

  2 228 4.3 Go tokenizer-go VS tokenizer

  Pure Go implementation of OpenAI's tiktoken tokenizer

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• tiktoken-go

  1 542 4.6 Go tokenizer-go VS tiktoken-go

  go version of tiktoken