[D] First glance at LLaMA

• llama-int8

  6 1,044 3.6 Python

  Quantized inference code for LLaMA models

To add a bit more context, the code other people linked (https://github.com/tloen/llama-int8) assumes single GPU. So if you want to run it on 2x3090, you'll need to modify it a bit:

• text-generation-webui

  876 36,293 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

I got the 13B model to work with RTX 3060 12GB and CPU + 64GB RAM. This repo: https://github.com/oobabooga/text-generation-webui

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project