Llama-tokenizer-js Alternatives

Similar projects and alternatives to llama-tokenizer-js

• chat-with-gpt

  39 2,261 5.3 TypeScript llama-tokenizer-js VS chat-with-gpt

  An open-source ChatGPT app with a voice

• Constrained-Text-Generation-Studio

  25 195 4.1 Python llama-tokenizer-js VS Constrained-Text-Generation-Studio

  Code repo for "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio" at the (CAI2) workshop, jointly held at (COLING 2022)

• SurveyJS

  surveyjs.io featured

  Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

  

• tiktoken

  30 9,800 6.7 Python llama-tokenizer-js VS tiktoken

  tiktoken is a fast BPE tokeniser for use with OpenAI's models.

  

• Constrained-Text-Genera

  11 - - llama-tokenizer-js VS Constrained-Text-Genera

• llama.go

  12 1,160 8.2 Go llama-tokenizer-js VS llama.go

  llama.go is like llama.cpp in pure Golang!

• nn-zero-to-hero

  10 10,396 2.4 Jupyter Notebook llama-tokenizer-js VS nn-zero-to-hero

  Neural Networks: Zero to Hero

• agency

  5 42 7.0 Go llama-tokenizer-js VS agency

  Agency: Robust LLM Agent Management with Go (by ryszard)

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• fastbpe

  1 2 5.5 Java llama-tokenizer-js VS fastbpe

  Java library implementing Byte-Pair Encoding Tokenization (by deepanprabhu)

• tokenizer

  2 228 4.3 Go llama-tokenizer-js VS tokenizer

  Pure Go implementation of OpenAI's tiktoken tokenizer

  

• gpt4-tokenizer-visualizer

  3 20 4.2 TypeScript llama-tokenizer-js VS gpt4-tokenizer-visualizer

  GPT4 Tokenizer Visualizer

• tiktoken

  1 592 7.6 Python llama-tokenizer-js VS tiktoken

  JS port and JS/WASM bindings for openai/tiktoken (by dqbd)

• gpt-tokenizer

  1 379 3.3 TypeScript llama-tokenizer-js VS gpt-tokenizer

  JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.