arxiv-sanity-lite VS LongLoRA

Once a week (at least!) some research group publishes another review paper to the cs.AI section on ArXiv. Look for new [papers with "survey" in the title](https://arxiv-sanity-lite.com/?q=survey&rank=time&tags=cs.AI...). You'll get surveys on every conceivable subtopic of ML/AI.

I use https://papers.labml.ai/papers/weekly/ to sort through compsci/math/machine learning papers and keep up, plus arxiv-sanity when it was still around (it is now https://arxiv-sanity-lite.com/).

Not AI based but pretty good for me. https://arxiv-sanity-lite.com/

Are you sure it is impossible?

https://arxiv-sanity-lite.com/?q=diffusion+image+editing&ran...

Arxiv sanity https://arxiv-sanity-lite.com/

LongAlpaca / One of many ways to extend context, and a useful dataset / https://arxiv.org/abs/2309.12307

Finally, LongLORA is a method to reduce the number of computations over a large context, and also specifically train the embed and norm layers fully, that is, no quantization or LORA for those. They are small layers and easy to train without too much VRAM cost, but the LongLORA authors noticed they have a big impact on long context performance. I am not using their computation reduction methods, but I am using their suggestion to train embed/norm layers fully.

That used to be true, but things like LongLORA and LongQLoRA demonstrate that you can increase the context length of a foundation model.

For reference, I am using the LongLoRA SFT implementation for fine-tuning a CodeLLaMA model on a code generation instruction. I have also attached my evaluation code below: