Show HN: LLaMA 3 tokenizer runs in the browser

• llama3-tokenizer-js

  1 18 7.5 JavaScript

  JS tokenizer for LLaMA 3

• next-token-prediction

  6 119 5.6 JavaScript

  Next-token prediction in JavaScript — build fast language and diffusion models.

Thanks for clarifying, this is exactly where I was confused.

I just read about how both sentencepiece and tiktoken tokenize.

Thanks for making this (in JavaScript no less!) and putting it online! I'm going to use it in my auto-completion library (here: https://github.com/bennyschmidt/next-token-prediction/blob/m...) instead of just `.split(' ')` as I'm pretty sure it will be more nuanced :)

Awesome work!

• SurveyJS

  surveyjs.io featured

  Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

  

Suggest a related project