gpt-2 VS gpt-neo

Medium models: Roughly between 1B to 10B parameters. This is where Mistral 7B, Phi-3, Gemma from Google DeepMind, and wizardlm2 sit. Fun fact: GPT 2 was a medium sized model, much smaller than its latest versions.

"Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper. "

They did release GPT-2 under the MIT License.

Did you arrive at this certainty through reading something other than what OpenAI has published? The document [0] that describes the training data for GPT-2 makes this assertion hilarious to me.

[0]: https://github.com/openai/gpt-2/blob/master/model_card.md#da...

I made a clone of https://github.com/openai/gpt-2 on my local laptop

By December 2020, EleutherAI had introduced The Pile, a comprehensive text dataset designed for training models. Subsequently, tech giants such as Microsoft, Meta, and Google used this dataset for training their models. In March 2021, they revealed GPT-Neo, an open-source model under Apache 2.0 license, which was unmatched in size at its launch. EleutherAI’s later projects include the release of GPT-J, a 6 billion parameter model, and GPT-NeoX, a 20 billion parameter model, unveiled in February 2022. Their work demonstrates the viability of high-quality open-source AI models.

Yeah, if that is your requirement you should definitely ignore chatterbot, as its older and probably not what your teacher wants. I'm looking at the gpt-neo docs right now: https://github.com/EleutherAI/gpt-neo

3.) EleutherAI's GPT-Neo and GPT-NeoX: EleutherAI is an independent research organization that aims to promote open research in artificial intelligence. They have released GPT-Neo, an open-source language model based on the GPT architecture, and are developing GPT-NeoX, a highly-scalable GPT-like model. You can find more information on their GitHub repositories: GPT-Neo: https://github.com/EleutherAI/gpt-neo GPT-NeoX: https://github.com/EleutherAI/gpt-neox

This is one of the first comprehensive plugins that has been rewritten to support multiple AI backends such as OpenAI GPT3+ and other custom sources in the future such as ChatGPT, GPT-J, GPT-neo and more.

GPT-Neo is newer than GPT-2 on the open source side of things. In my experience, it tends to give longer and more creative responses than GPT-2 but not on the level of GPT-3. I've not tried GPT-J or GPT-NeoX, but they're also open source and reportedly better than GPT-Neo (albeit less accessible).

Finally, yes-- it is trained via masked language modeling (text prediction). The approach has been fairly standard for years- the big difference with the GPT* models is the number of paramaters and volume of text-- we still haven't reached a ceiling with LLM parameters- they appear to keep improving with size. This training allows the model to learn a strong representation of language. Their training approach is published and open-source GPT* versions have already been made and released (https://github.com/EleutherAI/gpt-neo). However, the models are huge and can't be run locally for hobbyists. This gets at larger issues in democratization of ML.

- GPT-Neo 2.7B, released Mar/2021, and unmaintained/unsupported as of Aug/2021? or;