DeepSpeed VS mesh-transformer-jax

Filed a discussion item on the deepspeed project: https://github.com/microsoft/DeepSpeed/discussions/3531

Solution: I don't know; this is where I am stuck. https://github.com/microsoft/DeepSpeed/issues/1037 suggests that I just need to 'apt install libaio-dev', but I've done that and it doesn't help.

There could be some spectrum of this expertise. For instance, https://github.com/NVIDIA/FasterTransformer, https://github.com/microsoft/DeepSpeed

DeepSpeed Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales (https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat)

I'm following this closely, together with other efforts like GPTQ Quantization and Microsoft's DeepSpeed, all of which are bringing down the hardware requirements of these advanced AI models.

- https://github.com/microsoft/DeepSpeed

Anything that could bring this to a 10GB 3080 or 24GB 3090 without 60s/it per token?

Sure. Here's the repo I used for the fine-tuning: https://github.com/kingoflolz/mesh-transformer-jax. I used 5 epochs, and appart from that I kept the default parameters in the repo.

You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links

The pretrained model is already available.

I know nothing, but have heard Hugging Face is in that direction.

https://github.com/huggingface/transformers

>Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.

> These models can be applied on:

> - Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.

> - Images, for tasks like image classification, object detection, and segmentation.

> - Audio, for tasks like speech recognition and audio classification.

---

Also read about GPT-J, whose capability is comparable with GTP-3.

https://github.com/kingoflolz/mesh-transformer-jax

But I believe it requires buying or renting GPUs.

Well, many models hosted on Hugging Face were actually developed without HF Transformers first (and then were ported to HF Transformers by the community). It is the case with GPT-J. Here is the original GPT-J implementation: https://github.com/kingoflolz/mesh-transformer-jax

GPT-J with 6B parameters barelly scrapes by on a 16GB GPU (using KoboldAI, dunno what impact different scripts and stuff might have)

doesnt seem the code is there - pretrained models are there. https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b

https://huggingface.co/EleutherAI/gpt-j-6B

isnt that so ?

175B * 16 bits = 350GB, but it does compress a bit.

GPT-J-6B, which you can download at https://github.com/kingoflolz/mesh-transformer-jax, is 6B parameters but weighs 9GB. It does decompress to 12GB as expected. Assuming the same compression ratio, download size would be 263GB, not 350GB.

GPT-J

If you have a high-end PC, you might be able to run the Fairseq 13B model locally (it's labelled on that site as "dense_13b"). Alternatively, if you have a fairly powerful PC that can't quite run Fairseq 13B, you might still be able to run the GPT-J 6B model. You can also use the KoboldAI GPT-J 6B Google Colab if you can't run GPT-J 6B locally. While on the topic of KoboldAI, KoboldAI is also generally a pretty good frontend for running AI models locally. Back on the topic of GPT-J 6B, though, the EleutherAI GPT-J 6B demo is also a decent option for testing GPT-J 6B without any computing done on your device's end. It can also be used on mobile devices.