llama VS llama-dl

> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4.

LLaMa weights are f16s to start out with, no lowering necessary to get to there.

You can stream weights from RAM to the GPU pretty efficiently. If you have >= 32GB ram and >=2GB vram my code here should work for you: https://github.com/gmorenz/llama/tree/gpu_offload

There's probably a cleaner version of it somewhere else. Really you should only need >= 16 GB ram, but the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously.

My code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd

Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056...

At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at once on the same GPU) is a good as it gets when you need to read the whole model from disk each token.

At a 125GB and a 2MB/s read (largest model, what I get from my ssd) that's 60 seconds per token (1 day per 1440 words), which isn't exactly practical. Which is really the issue here, if you need to stream the model from an SSD because you don't have enough RAM, it is just a fundamentally slow process.

You could probably optimize quite a bit for batch throughput if you're ok with the latency though.

I don't know about this fork specifically, but in general yes absolutely.

Even without enough ram, you can stream model weights from disk and run at [size of model/disk read speed] seconds per token.

I'm doing that on a small GPU with this code, but it should be easy to get this working with the CPU as compute instead (and at least with my disk/CPU, I'm not even sure that it would run even slower, I think disk read would probably still be the bottleneck)

https://github.com/gmorenz/llama/tree/ssd

There seems to be some confusion here. Let me step in as someone who has gone through this.

My repo https://github.com/shawwn/llama-dl was taken down last March by Facebook. They asserted copyright over LLaMA, which is obviously bogus since it was trained on data they do not own the copyright to. I was bummed about this, but after I mentioned on HN that I was willing to fight Meta, an anonymous person named L contacted me and sent $20k of Monero to cover legal fees. I was also contacted by an amazing lawyer who wanted to represent me in this. I was absurdly fortunate on both counts.

He drafted a counternotice, we sent it, and then my repo was restored within a week or so.

GitHub had no choice in the matter. Legally this is a required process. Ditto for GitLab. Both are US companies.

When YouTube-dl was taken down some time ago by a DMCA, Nat went to bat and got it restored, and GitHub made some sort of pledge to cover legal fees associated with bogus takedown requests.

Here’s the shitty part for this particular situation. A case can be made that the emulator is for the purpose of circumventing copyright protection mechanisms. This, sadly, is a solid legal basis for issuing a lawful takedown, as much as we all absolutely despise that idea. It’s pretty clear cut; Nintendo doesn’t want Switch games to be run on non-Nintendo platforms, and the emulator seeks to enable Switch games to be run on any platform. Therefore, the intent of the emulator is to circumvent Nintendo’s protection mechanisms.

So where does this leave us? Well, the team can file a counternotice. GitLab will restore the repo. But that opens up the team to a lawsuit by Nintendo. And as much as I want to stand up to bullies, there’s a difference between standing up to a guy shoving a kid in a locker vs standing up to a Silverback gorilla charging at you. Nintendo’s legal history implies the latter.

Welcome to Nintendo pain. The Smash community has been dealing with Nintendo’s BS for decades now. They shut down tournaments that use emulators for Smash Melee. And no one can do anything, because it’s their legal right to do so.

Your philosophical argument is interesting, but what the op was saying was one of the linked repos in inaccessible due to DMCA: https://github.com/shawwn/llama-dl

So while what you say may be true the DMCA seems to have worth for these orgs because they can get code removed by the host, who is uninterested in litigating, and the repo owner likely is even less capable of litigating the DMCA.

Unfortunately as a tool of fear and legal gridlock DMCA has shown itself to be very useful to those with ill intent.

Here are some useful links: https://github.com/shawwn/llama-dl and https://rentry.org/llama-tard-v2#tips-and-tricks

Sure. You can get models with magnet link from here https://github.com/shawwn/llama-dl/

To get running, just follow these steps https://github.com/ggerganov/llama.cpp/#usage

https://rentry.org/llama-tard-v2#tips-and-tricks https://github.com/shawwn/llama-dl