LaMDA-rlhf-pytorch VS PaLM-rlhf-pytorch

I made a mistake about the LaMDa, what I mean is the machine learning model that Lamda based on: Transformer. Though you can still access to the paper: [2201.08239] LaMDA: Language Models for Dialog Applications (arxiv.org) and a pyTorch implementation here: conceptofmind/LaMDA-rlhf-pytorch: Open-source pre-training implementation of Google's LaMDA in PyTorch. Adding RLHF similar to ChatGPT. (github.com)

I know it'll be 10 times worse for now at best since this is refined proprietary tech where a lot of resources went into but I'm still looking forward to something aiming to achieve this level of fluency and context understanding that can be used open and freely. Closest thing I could find so far might be GPT2 or laMDA?

An open-source implementation for the pre-training architecture of Google's LaMDA in PyTorch. The research paper outlines an autoregressive, decoder-only, GPT-like transformer language model. The transformer uses T5 relative positional bias in the attention layers and gated-GELU activation function in the feed-forward layers.

The repository currently contains a script for basic training as well as Huggingface datasets and Weights & Biases integration.

LaMDA research paper: https://arxiv.org/abs/2201.08239

Github repository for the model: https://github.com/conceptofmind/LaMDA-pytorch

The pre-training architecture was peer-reviewed by Dr. Phil Wang. Please check out and support his work: https://github.com/lucidrains.

Updates: https://twitter.com/EnricoShippole

If you want to view the pre-training architecture of a transformer model such as LaMDA, the one you are referencing in this post, you can view that here: https://github.com/conceptofmind/LaMDA-pytorch/blob/main/lamda_pytorch/lamda_pytorch.py

An example for an iterable data loader like this would be: https://github.com/conceptofmind/LaMDA-pytorch/blob/main/dataloader/stream_dataloader.py

Github repository for the model: https://github.com/conceptofmind/LaMDA-pytorch

ChatGPT isn't open sourced so we don't know what the actual implementation is. I think you can read Open Assistant's source code for application design. If that is too much, try Open Chat Toolkit's source code for developer tools . If you need very bare implementation, you should go for lucidrains/PaLM-rlhf-pytorch.

AFAIK, it is not. They are using the open-source re-implementation of Phil Wang (aka lucidrains), which is available here: https://github.com/lucidrains/PaLM-rlhf-pytorch

Not sure what do you mean by putting source code in double quote, but I don't think the source code is petabytes of text. GPT-2 implementation is few hundred lines of Python (in HuggingFace). PaLM + RLHF - Pytorch (Basically ChatGPT but with PaLM) is less than 1000 lines.

GitHub Link: https://github.com/lucidrains/PaLM-rlhf-pytorch

Well... Not necessarily. Actually, if you want to be extra thrifty, you could even go without an ML expert. Just use an open-source one, like LaMDA or PaLM. After that, use chatGPT to build you a basic front end (which would still be better than CAI lol).

and PaLM with RLHF from Phil Wang (open model, needs to be trained): https://github.com/lucidrains/PaLM-rlhf-pytorch

Closest you can get is probably with Google T5-Flan [1].

It is not the size of the model or the text it was trained on that makes ChatGPT so performant. It is the additional human assisted training to make it respond well to instructions. Open source versions of that are just starting to see the light of day [2].

[1] https://huggingface.co/google/flan-t5-xxl

[2] https://github.com/lucidrains/PaLM-rlhf-pytorch