GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3 (by latitudegames)
Add to my DEV experience Suggest topics
Source Code
GPT-3-Encoder Reviews
Suggest alternative
Edit details
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
surveyjs.io
Sponsored
Suggest an alternative to GPT-3-Encoder

GPT-3-Encoder reviews and mentions

Posts with mentions or reviews of GPT-3-Encoder. We have used some of these posts to build our list of alternatives and similar projects.
  • Mastering GPT-3: The mathematics of logprobs for Ruby Devs
    1 project | dev.to | 4 Feb 2023
    A full list of GPT tokens can be found at the following link: https://github.com/latitudegames/GPT-3-Encoder/blob/master/vocab.bpe
  • token optimization tools
    1 project | /r/NovelAi | 29 Jun 2021
    ... but both OpenAI's web tokenizer, and NovelAI (at least based on its displayed token counts), seem to use Latitude's JavaScript port which tokenizes 'll wrong, as two tokens. The model was trained on input that was processed with transformers.GPT2TokenizerFast.from_pretrained('gpt2') and so expects it to be just one token, with token ID 1183. I pointed this out to NovelAI's devs... on the most random channel I could think of to do it, so they might completely miss it/forget to handle it... but oh well, we'll see.

Stats

Basic GPT-3-Encoder repo stats
2
711
0.0
about 1 year ago

latitudegames/GPT-3-Encoder is an open source project licensed under MIT License which is an OSI approved license.

The primary programming language of GPT-3-Encoder is JavaScript.


Sponsored
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com