Running AI Models on NixOS

• text-generation-webui

  876 36,552 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

• nix-stable-diffusion

  2 107 5.5 Nix

  Flake for running SD on NixOS

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• koboldcpp-rocm

  3 278 10.0 C++

  AI Inferencing at the Edge. A simple one-file way to run various GGML models with KoboldAI's UI with AMD ROCm offloading

• flake

  5 593 4.4 Nix

  A Nix flake for many AI projects

  

• llama.cpp

  775 57,463 10.0 C++

  LLM inference in C/C++

Llama.cpp has its own flake, and I've had good luck getting it to build on both Arch (with rocm) and macOS (with metal). Just clone, cd into the directory, and nix build. Then you can use the llama and llama-server binaries in result/bin.

• gpt4all-nix

  1 37 10.0 Nix

  Discontinued Nix flake for gpt4all

I'm (whenever I find the time) maintaining a gpt4all Flake for NixOS: https://github.com/polygon/gpt4all-nix/

Suggest a related project