A comprehensive guide to running Llama 2 locally

1. llama.cpp

   1 1,087 126,810 10.0 C++

   LLM inference in C/C++

   

   For my fellow Windows shills, here's how you actually build it on windows:

   Before steps:

   1. (For Nvidia GPU users) Install cuda toolkit https://developer.nvidia.com/cuda-downloads

   2. Download the model somewhere: https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolv...

   In Windows Terminal with Powershell:

       git clone https://github.com/ggerganov/llama.cpp

2. Kargo

   akuity.io sponsored

   Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.

   

3. textgen

   2 888 47,600 9.7 Python

   Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.

   

   The bash script is downloading llama.cpp, a project which allows you to run LLaMA-based language models on your CPU. The bash script then downloads the 13 billion parameter GGML version of LLaMA 2. The GGML version is what will work with llama.cpp and uses CPU for inferencing. There are ways to run models using your GPU, but it depends on your setup whether it will be worth it.

   I would highly recommend looking into the text-generation-webui project (https://github.com/oobabooga/text-generation-webui). It has a one-click installer and very comprehensive guides for getting models running locally and where to find models. The project also has an "api" command flag to let you use it like you might use a web-based service currently.

4. ollama

   3 800 180,166 9.8 Go

   Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

   

   > adjust your paths as necessary. It has a tendency to talk to itself.

   This is always a fun surprise. What I've seen help, especially with chat models, is to use a prompt template. Some tools (e.g. https://ollama.ai/ – disclosure: I'm a maintainer) use a default, model-specific prompt template when you run the model. This is easier for users since they can just input your chat messages and get answers. Every model is trained differently.

   But with llama.cpp it you'd need wrap your prompt text with the right template. For llama 2, the facebook developers' generation code wraps the system prompt and user prompts in specific tags (<>{system prompt} and [INST]{user prompt}[/INST]) respectively): https://github.com/facebookresearch/llama/blob/main/llama/ge....

   Worth noting that customizing prompt templates can be fun – you don't have to use them – I've had a model generate a conversation between a few characters that talk to each other for example – it's pretty entertaining!

5. llama

   4 190 59,528 4.7 Python

   Inference code for Llama models

   

   > adjust your paths as necessary. It has a tendency to talk to itself.

   This is always a fun surprise. What I've seen help, especially with chat models, is to use a prompt template. Some tools (e.g. https://ollama.ai/ – disclosure: I'm a maintainer) use a default, model-specific prompt template when you run the model. This is easier for users since they can just input your chat messages and get answers. Every model is trained differently.

   But with llama.cpp it you'd need wrap your prompt text with the right template. For llama 2, the facebook developers' generation code wraps the system prompt and user prompts in specific tags (<>{system prompt} and [INST]{user prompt}[/INST]) respectively): https://github.com/facebookresearch/llama/blob/main/llama/ge....

   Worth noting that customizing prompt templates can be fun – you don't have to use them – I've had a model generate a conversation between a few characters that talk to each other for example – it's pretty entertaining!

6. text-generation-inference

   5 34 10,812 8.8 Python

   Discontinued Large Language Model Text Generation Inference

   

   MLC LLM (iOS/Android)

   Which is not really comprehensive... If you have a linux machine with GPUs, i'd just use hugging face inference (https://github.com/huggingface/text-generation-inference). And I am sure there are other things that could be covered.

7. llm

   6 104 12,466 9.6 Python

   Access large language models from the command-line

   

   My LLM command line tool can do that - it logs everything to a SQLite database and has an option to continue a conversation: https://llm.datasette.io

8. blog

   7 5 3,512 9.5 Jupyter Notebook

   Public repo for HF blog posts (by huggingface)

   

   If you just want to do inference/mess around with the model and have a 16GB GPU, then this[0] is enough to paste into a notebook. You need to have access to the HF models though.

   0. https://github.com/huggingface/blog/blob/main/llama2.md#usin...

9. AppSignal

   www.appsignal.com sponsored

   Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.

   

10. llama

    8 2 77 6.2 Python

    Inference code for LLaMA models on CPU and Mac M1/M2 GPU (by krychu)

    

    Self-plug. Here’s a fork of the original llama 2 code adapted to run on the CPU or MPS (M1/M2 GPU) if available:

    https://github.com/krychu/llama

    It runs with the original weights, and gets you to ~4 tokens/sec on MacBook Pro M1 with the 7B model.

11. Pytorch

    9 425 102,755 10.0 Python

    Tensors and Dynamic neural networks in Python with strong GPU acceleration

    

12. DeepSpeed

    10 54 43,061 9.6 Python

    DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

    

13. bitsandbytes

    11 63 8,460 9.2 Python

    Accessible large language models via k-bit quantization for PyTorch.

    

14. stable-diffusion-webui

    12 2,815 164,802 0.0 Python

    Stable Diffusion web UI

    

Suggest a related project