LLaMA-Adapter VS bench-warmers

Either LLaMA with Alpaca LoRA 65B, or LLaMA-Adapter-V2-65B chat demo. I haven't seen any tests of the 65B LLaMA-Adapter-V2, but they claim it's as good as ChatGPT when compared using GPT-4.

Chat65B: https://github.com/ZrrSkywalker/LLaMA-Adapter/tree/main/llama_adapter_v2_chat65b

How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at https://github.com/ZrrSkywalker/LLaMA-Adapter.

Quick Read: https://www.marktechpost.com/2023/03/31/meet-llama-adapter-a-lightweight-adaption-method-for-fine-tuning-instruction-following-llama-models-using-52k-data-provided-by-stanford-alpaca/ Paper: https://arxiv.org/pdf/2303.16199.pdf Github: https://github.com/ZrrSkywalker/LLaMA-Adapter

Found relevant code at https://github.com/ZrrSkywalker/LLaMA-Adapter + all code implementations here

i have more ideas than I know what to do with, help yourself: https://github.com/dmarx/bench-warmers

you're welcome to help yourself to my ideas (no guarantees that they're any good or even comprehensible, I do a lot of my brainstorming while high). here's my brainstorming space, scroll down for a categorized ToC: https://github.com/dmarx/bench-warmers

I've decided to just lean into it and am literally just giving my ideas away. https://github.com/dmarx/bench-warmers

it might sound like a lot, but it's actually really lightweight and easy to use. Check it out: https://github.com/dmarx/bench-warmers

you've sort of become victims of your own success: as another pandas dev mentioned, you want to preserve backwards compatibility and this significantly complicates any restructuring. I'm sympathetic and am not sure what the best solution here would be. I had this idea last night but i'm not sure I like this approach either.

To be clear, i'm talking about e.g. vision impaired, hearing impaired, etc. Here's an example of a project idea in this space (possibly a bit more ambitious than what you're looking for but if you think you could tackle this I encourage you to take a stab at it): https://github.com/dmarx/bench-warmers/blob/main/automated-video-description.md