gptqlora VS datablations

GPTQLoRA: Efficient Finetuning of Quantized LLMs with GPTQ (https://github.com/qwopqwop200/gptqlora/tree/main)

The difference from QLoRA is that GPTQ is used instead of NF4 (Normal Float4) + DQ (Double Quantization) for model quantization. The advantage is that you can expect better performance because it provides better quantization than conventional bitsandbytes. The downside is that it is a one-shot quantization methodology, so it is more inconvenient than bitsandbytes, and unlike bitsandbytes, it is not universal. I'm still experimenting, but it seems to work. At least, I hope it can be more options for people using LoRA. https://github.com/qwopqwop200/gptqlora/tree/main

1x chinchilla means it's not really undertrained but that more could be squeezed without excessive difficulty https://arxiv.org/abs/2305.16264

You might want to give a read to "Scaling Data-Constrained Language Models" [1]. They basically generalized the Chinchilla scaling law by investigating behavior on multi-epoch runs.

[1] https://arxiv.org/abs/2305.16264

I would imagine that there is a lot of overlap, yeah. That said, training on repeated data does seem to be effective at this level.

Scaling Data-Constrained Language Models (https://arxiv.org/abs/2305.16264)