flake
koboldcpp-rocm
| flake | koboldcpp-rocm | |
|---|---|---|
| 5 | 3 | |
| 593 | 278 | |
| 3.9% | - | |
| 4.4 | 10.0 | |
| 7 days ago | 3 days ago | |
| Nix | C++ | |
| GNU Affero General Public License v3.0 | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
flake
- Running AI Models on NixOS
- Nixified.Ai Release 2
-
Llama.cpp: Full CUDA GPU Acceleration
> Ideally, there's Nix (and poetry2nix) that could take care of everything, but only a few folks write Flakes for their projects.
Relevant to "AI, Python, setting up is hard ... nix", there's stuff like:
https://github.com/nixified-ai/flake
-
Can you substitute conda with Nix for Data Science and ML/AI?
However, I would reach out to the Nixified.ai folks about it, because I can see that the invoke.ai build script mentions pytorch and several other hard-to-install packages (albeit not detectron).
- A Nix flake for many AI projects
koboldcpp-rocm
- Running AI Models on NixOS
-
Show HN: Ollama for Linux – Run LLMs on Linux with GPU Acceleration
This one is basically SOTA for AMD, if you can install rocm properly:
https://github.com/YellowRoseCx/koboldcpp-rocm
- KoboldCPP now has (experimental) RX 6700XT/gfx1031 support
What are some alternatives?
nonguix - Nonguix mirror – pull requests ignored, please use upstream for that
llama.cpp - LLM inference in C/C++
guix-nonfree - Unofficial collection of packages that are not going to be accepted in to guix
cody - AI that knows your entire codebase
lit-llama - Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
Dumbar - A smrt, no, smart, ok, no dumb smartbar for Ollama
llama_cpp.rb - llama_cpp provides Ruby bindings for llama.cpp
gpt4all-nix - Nix flake for gpt4all
serving - A flexible, high-performance serving system for machine learning models
guix-nonfree
TokenHawk - WebGPU LLM inference tuned by hand [Moved to: https://github.com/kayvr/token-hawk]
llama.cpp - Port of Facebook's LLaMA model in C/C++