trl VS OutRun

I am using TRL for RLHF fine-tuning the Llama-2-7B model and getting an OOM error (even with batch_size=1). If anyone used TRL for RLHF can please tell me what I am doing wrong? Code details can be found in the GitHub issue.

SFTtrainer from huggingface

For lora - PEFT seems to work. I don't have patience to wait 5 hours, but modifying this example seems to work. You don't even need to modify that much, as their model just as neo-x uses query_key_value name for self-attention.

They have examples of making GPT output more positive (code) by using a sentiment model as reward. There are other examples about reducing toxicity, summarization here: https://github.com/lvwerra/trl/tree/main/examples . Should be fairly simple to modify the sentiment example and try the calculator reward you mentioned above.

You can use this -> https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt-neox-20b_peft/merge_peft_adapter.py

Just the hh directly. From the results it seems like it might possibly be enough but I might also try instruction tuning then running the whole process from that base. I will also be running the reinforcement learning by using a Lora using this as an example https://github.com/lvwerra/trl/tree/main/examples/sentiment/scripts/gpt-neox-20b_peft

This package is pretty simple to use! https://github.com/lvwerra/trl

Ive had a similar problem to you, but I have mostly solved it by using the Outrun app (which is free and open source), and going into the settings and turning up the GPS smoothing. I mostly run in central london, and it seems to work with the tall buildings here.

https://github.com/timfraedrich/OutRun

https://github.com/timfraedrich/OutRun/issues/91

> I wouldn't necessarily say abandoned. I still work on it from time to time, but progress is very very slow and I cannot prioritise it over other things atm

Shout out to the app "Out-Run" that I use to track my rides. Privacy focused way to track your rides without all of the bullshit social media that comes with Strava. No account, all data stored locally, can easily export your rides, no data collected. Have exchanged a few emails with the dev and he's a super cool dude, so I shout out his app at every opportunity. All open source as well.

This is the GitHub page.