trlx VS summarize-from-feedback

Transformer Reinforcement Learning X on GitHub

We drove and released the top RLHF framework TRLX for example from our Carper AI lab used by some of the biggest companies in the world: https://github.com/CarperAI/trlx

If you checkout the trlx repo they have some examples and they have an example of how they trained sft and ppo on the hh dataset. So it’s basically that but with llama. https://github.com/CarperAI/trlx/blob/main/examples/hh/sft_hh.py

We’ll release our first trained model with Stability AI soon. If you want to start tinkering with RLHF now, we’re also helping develop TRLX: https://github.com/CarperAI/trlx — the open source library for reinforcement learning with transformers.

trlX library here: https://github.com/CarperAI/trlx

Note that they released code, models and raw data here: https://github.com/openai/summarize-from-feedback

Generating well-written human text answering specific prompts is very costly, as it often requires hiring part-time staff (rather than being able to rely on product users or crowdsourcing). Thankfully, the scale of data used in training the reward model for most applications of RLHF (~50k labeled preference samples) is not as expensive. However, it is still a higher cost than academic labs would likely be able to afford. Currently, there only exists one large-scale dataset for RLHF on a general language model (from Anthropic) and a couple of smaller-scale task-specific datasets (such as summarization data from OpenAI). The second challenge of data for RLHF is that human annotators can often disagree, adding a substantial potential variance to the training data without ground truth.

Found relevant code at https://github.com/openai/summarize-from-feedback + all code implementations here

> Software 2.0 is happening right now. GTP-3 and Tesla FSD are examples of this.

I agree with this. As an anecdote, I've spent the past decade explaining to clients that things like natural language question answering and abstractive summarization are impossible, and now we have OpenAI and others dropping pretrained models like https://github.com/openai/summarize-from-feedback that turn all those assumptions on their head. There are caveats, of course, but I've gone from a deep learning skeptic (I started my career with "traditional" ML and NLP) to believing that these sorts of techniques are truly revolutionary and we are only yet scratching the surface of what's possible with them.