LongLoRA VS LLM-Finetuning-Hub

LongAlpaca / One of many ways to extend context, and a useful dataset / https://arxiv.org/abs/2309.12307

Finally, LongLORA is a method to reduce the number of computations over a large context, and also specifically train the embed and norm layers fully, that is, no quantization or LORA for those. They are small layers and easy to train without too much VRAM cost, but the LongLORA authors noticed they have a big impact on long context performance. I am not using their computation reduction methods, but I am using their suggestion to train embed/norm layers fully.

That used to be true, but things like LongLORA and LongQLoRA demonstrate that you can increase the context length of a foundation model.

For reference, I am using the LongLoRA SFT implementation for fine-tuning a CodeLLaMA model on a code generation instruction. I have also attached my evaluation code below:

Hi everyone, we've been working on benchmarking different open-source LLMs. We measure, in particular, on the performance of these models once finetued (via QLoRA) on classic NLP downstream tasks like summarization and classification. We also put particular emphasis on benchmarking inference time/cost for these models once deployed.

Hello all,

I have been working on benchmarking different LLMs -- both open-source and closed-source.

Repo: https://github.com/georgian-io/LLM-Finetuning-Hub

Precisely, I am comparing their out-of-the-box capabilities (prompting) and their fine-tuned conterparts!

So far, the following models have been benchmarked:

Open-Source:

- Finetuning with QLoRA

I evaluate how Falcon does on classification tasks when compared to Bert and Distilbert.

Moreover, I talk about different ways you can deploy the model, and the associated costs!

The code for all of my experiments are available on: https://github.com/georgian-io/LLM-Finetuning-Hub

Happy reading and learning!