PaLM-rlhf-pytorch VS DALLE-mtf

ChatGPT isn't open sourced so we don't know what the actual implementation is. I think you can read Open Assistant's source code for application design. If that is too much, try Open Chat Toolkit's source code for developer tools . If you need very bare implementation, you should go for lucidrains/PaLM-rlhf-pytorch.

AFAIK, it is not. They are using the open-source re-implementation of Phil Wang (aka lucidrains), which is available here: https://github.com/lucidrains/PaLM-rlhf-pytorch

Not sure what do you mean by putting source code in double quote, but I don't think the source code is petabytes of text. GPT-2 implementation is few hundred lines of Python (in HuggingFace). PaLM + RLHF - Pytorch (Basically ChatGPT but with PaLM) is less than 1000 lines.

GitHub Link: https://github.com/lucidrains/PaLM-rlhf-pytorch

Well... Not necessarily. Actually, if you want to be extra thrifty, you could even go without an ML expert. Just use an open-source one, like LaMDA or PaLM. After that, use chatGPT to build you a basic front end (which would still be better than CAI lol).

and PaLM with RLHF from Phil Wang (open model, needs to be trained): https://github.com/lucidrains/PaLM-rlhf-pytorch

Closest you can get is probably with Google T5-Flan [1].

It is not the size of the model or the text it was trained on that makes ChatGPT so performant. It is the additional human assisted training to make it respond well to instructions. Open source versions of that are just starting to see the light of day [2].

[1] https://huggingface.co/google/flan-t5-xxl

[2] https://github.com/lucidrains/PaLM-rlhf-pytorch

This vision is in line with EleutherAI, a non-profit organization founded in July 2020 by a group of researchers. Driven by the perceived opacity and the challenge of reproducibility in AI, their goal was to create leading open-source language models.

My current idea is to utilize the EleutherAI pythia (Databricks Dolly). I would like to know whether translating the Dolly-15k dataset into the desired language using state-of-the-art translation techniques like DeepL would be a viable approach to fine-tune the Pythia base model. I want to use this model for semantic search, so perfection is not a necessity.

There are open source AI efforts, like EleutherAI. Needless to say, they are lagging behind big players, but it's better than nothing.

The new GPT 3.5 isn't aware what are GPT-3.5 or davinci-002 (repeatable) and claimed that it was designed by EleutherAI and has only 6 bil parameters (wasn't been able to repeat but didn't really try).

Hi, my name is Stella Biderman and I run EleutherAI, the one of the foremost non-profit research institutes in the world that trains and studies large language models. I have been involved with the majority of models to hold the title “largest open source GPT model in the world” and have dabbled in exploring using plagiarism detection tools to identify code written by GPT-J.

dolly-v2-12bis a 12 billion parameter causal language model created by Databricks that is derived from EleutherAI’s Pythia-12b and fine-tuned on a ~15K record instruction corpus generated by Databricks employees and released under a permissive license (CC-BY-SA)

It is true that Emad needs to find an appropriate business model. The good news is that the hype is still undergoing. I'm sure that Emad can grab another round of liquidity injection. He got plenty of resources. Remember he is also from the finance industry. He got https://www.eleuther.ai/ which can supply a secured, in-house custom LLM equivalent to bloombergGPT.

By promoting fully open-source AI, i.e. making datasets, models, methodology and codebases freely available and transparent. What OpenAI claimed to be aiming for, basically.