LongLoRA
discus
| LongLoRA | discus | |
|---|---|---|
| 4 | 1 | |
| 2,478 | 60 | |
| 3.8% | - | |
| 9.1 | 7.7 | |
| 3 months ago | 6 months ago | |
| Python | Python | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
LongLoRA
-
Ask HN: AI/ML papers to catch up with current state of AI?
LongAlpaca / One of many ways to extend context, and a useful dataset / https://arxiv.org/abs/2309.12307
-
Aurelian: 70B 32K story-writing (and more) [Alpha]
Finally, LongLORA is a method to reduce the number of computations over a large context, and also specifically train the embed and norm layers fully, that is, no quantization or LORA for those. They are small layers and easy to train without too much VRAM cost, but the LongLORA authors noticed they have a big impact on long context performance. I am not using their computation reduction methods, but I am using their suggestion to train embed/norm layers fully.
-
Why train on Yi 4K instead of 200K?
That used to be true, but things like LongLORA and LongQLoRA demonstrate that you can increase the context length of a foundation model.
-
Using Overfitting to Debug My LLM [P]
For reference, I am using the LongLoRA SFT implementation for fine-tuning a CodeLLaMA model on a code generation instruction. I have also attached my evaluation code below:
discus
What are some alternatives?
relora - Official code for ReLoRA from the paper Stack More Layers Differently: High-Rank Training Through Low-Rank Updates
kani - kani (カニ) is a highly hackable microframework for chat-based language models with tool use/function calling. (NLP-OSS @ EMNLP 2023)
Zicklein - Finetuning instruct-LLaMA on german datasets.
tdk-demo - This is a collection of TDK demo projects that use different databases and options
torch-adapters - Small Library of PyTorch Adaptation modules
tiger - Open Source LLM toolkit to build trustworthy LLM applications. TigerArmor (AI safety), TigerRAG (embedding, RAG), TigerTune (fine-tuning)
punica - Serving multiple LoRA finetuned LLM as one
distilabel - ⚗️ distilabel is a framework for synthetic data and AI feedback for AI engineers that require high-quality outputs, full data ownership, and overall efficiency.
RingAttention - Transformers with Arbitrarily Large Context
start-llms - A complete guide to start and improve your LLM skills in 2024 with little background in the field and stay up-to-date with the latest news and state-of-the-art techniques!
xTuring - Build, customize and control you own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6
LLM-Adapters - Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"