kener VS trl

1. Kener

I think the incident.svelte file could use some love, is it best practice to put part of the phrase somewhere else? Doesn't it increase the cognitive stress? Like there is somewhere a phrase, but part of the text is being calculated https://github.com/rajnandan1/kener/blob/74ea57d6bbf6ac4dd3e...

Isn't it easier to understand what is going on just by calculating the condition on the top and put the text on the markup based on that condition?

I feel like there are few places where in order to don't duplicate part of the text it's being made extremely difficult what the text is going to be by putting it far away

I am using TRL for RLHF fine-tuning the Llama-2-7B model and getting an OOM error (even with batch_size=1). If anyone used TRL for RLHF can please tell me what I am doing wrong? Code details can be found in the GitHub issue.

SFTtrainer from huggingface

For lora - PEFT seems to work. I don't have patience to wait 5 hours, but modifying this example seems to work. You don't even need to modify that much, as their model just as neo-x uses query_key_value name for self-attention.

They have examples of making GPT output more positive (code) by using a sentiment model as reward. There are other examples about reducing toxicity, summarization here: https://github.com/lvwerra/trl/tree/main/examples . Should be fairly simple to modify the sentiment example and try the calculator reward you mentioned above.

You can use this -> https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt-neox-20b_peft/merge_peft_adapter.py

Just the hh directly. From the results it seems like it might possibly be enough but I might also try instruction tuning then running the whole process from that base. I will also be running the reinforcement learning by using a Lora using this as an example https://github.com/lvwerra/trl/tree/main/examples/sentiment/scripts/gpt-neox-20b_peft

This package is pretty simple to use! https://github.com/lvwerra/trl