empirical-philosophy VS alpaca_lora_4bit

One way that I've been framing this in my head (and in an application I'm building) is that gpt-3 will be useful for analytic tasks where as gpt-4 will be required for synthetic tasks. I'm using "analytic" and "synthetic" in the same way as in this writeup https://github.com/williamcotton/empirical-philosophy/blob/m...

Hey Simon! I've been digging your writings on LLMs lately.

I've been having some decent luck with some of the approaches that I've discussed in the following articles and projects:

From Prompt Alchemy to Prompt Engineering: An Introduction to Analytic Augmentation: https://github.com/williamcotton/empirical-philosophy/blob/m...

https://www.williamcotton.com/articles/writing-web-applicati...

https://github.com/williamcotton/transynthetical-engine

I'd love to hear your thoughts on the matter!

You’re not actually doing any research.

Here is my research: https://github.com/williamcotton/empirical-philosophy/blob/m...

It is clear that analytic augmentations will result in more factual information.

Your claims are unfounded and untested.

Take a look at

https://github.com/williamcotton/empirical-philosophy/blob/m...

https://langchain.readthedocs.io/en/latest/

They can be taught!

I've been developing a methodology around prompt engineering that I have found very useful:

https://github.com/williamcotton/empirical-philosophy/blob/m...

A few more edits and it's ready for me to submit to HN and then get literally no further attention!

That's not a rebuttable of a claim that Bing is more accurate.

A proper rebuttable would involve empirical evidence that Bing is no more accurate than other LLM tools that do not add analytical augmentations such as search results to their prompts.

Based on empirical evidence, I find that analytical augmentations do indeed result in more accurate results:

https://github.com/williamcotton/empirical-philosophy/blob/m...

For training there is also https://github.com/johnsmith0031/alpaca_lora_4bit

I've been trying to fine tune an erotica model on some large context chat history (reverse proxy logs) and a literotica-instruct dataset I made, with a max context of 8k. The large context size eats a lot of VRAM so I've been trying to find the most efficient way to experiment considering I'd like to do multiple runs to test some ideas. So I'm going to try and use https://github.com/johnsmith0031/alpaca_lora_4bit, which is supposed to train faster and use less memory than qlora.

Follow up the popular work of u/tloen alpaca-lora, I wrapped the setup of alpaca_lora_4bit to add support for GPTQ training in form of installable pip packages. You can perform training and inference with multiple quantizations method to compare the results.

" Using LoRAs with GPTQ-for-LLaMa This requires using a monkey patch that is supported by this web UI: https://github.com/johnsmith0031/alpaca_lora_4bit"

4-bit GPTQ LoRA training was available since early April. I did not see any comparison to it in the QLoRA paper or even a mention, so it makes me think they were not aware it already existed.

https://github.com/johnsmith0031/alpaca_lora_4bit is still the SOTA - Faster than qlora, trains on a GPTQ base.

I would also like to know how one would finetune this in 4 bit? I think one could take the merged 8K PEFT with the LLaMA weights, and then quantize it to 4 bit, and then train with https://github.com/johnsmith0031/alpaca_lora_4bit ?

I was under the impression that you just git clone this repo into text-generation-webui/repositories (so you would have GPTQ_for_Llama and alpaca_lora_4bit in the folder), and then just load with monkey patch. Is that not correct? I also tried just downloading alpaca_lora_4bit on its own, git cloning text-gen-webui within it, and installing requirements.txt for both and running with monkey patch. I was following the sections of alpaca_lora_4bit, "Text Generation Webui Monkey Patch" and "monkey patch inside webui"

I dont have any familiarity with esxi, but I can say that there are quite a few posts about people doing it on proxmox. I've currently got a machine with 2x3090 passing through to VM's. When I'm training, I pass them both through to the same VM and can do lora 4-bit training on llama33 using https://github.com/johnsmith0031/alpaca_lora_4bit. Then, at inference time, I run a single card into a different VM, and have an extra card available for experimentation.