llama.cpp
rust-gpu
Our great sponsors
llama.cpp | rust-gpu | |
---|---|---|
769 | 82 | |
55,846 | 6,930 | |
- | 1.9% | |
10.0 | 8.2 | |
4 days ago | 6 days ago | |
C++ | Rust | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
llama.cpp
-
Phi-3 Weights Released
well https://github.com/ggerganov/llama.cpp/issues/6849
- Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding
- Llama.cpp Working on Support for Llama3
-
Embeddings are a good starting point for the AI curious app developer
Have just done this recently for local chat with pdf feature in https://recurse.chat. (It's a macOS app that has built-in llama.cpp server and local vector database)
Running an embedding server locally is pretty straightforward:
- Get llama.cpp release binary: https://github.com/ggerganov/llama.cpp/releases
- Mixtral 8x22B
- Llama.cpp: Improve CPU prompt eval speed
-
Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
Ah, thanks for this! I can't edit my parent comment that you replied to any longer unfortunately.
As I said, I only compared the contributors graphs [0] and checked for overlaps. But those apparently only go back about year and only list at most 100 contributors ranked by number of commits.
[0]: https://github.com/ollama/ollama/graphs/contributors and https://github.com/ggerganov/llama.cpp/graphs/contributors
-
KodiBot - Local Chatbot App for Desktop
KodiBot is a desktop app that enables users to run their own AI chat assistants locally and offline on Windows, Mac, and Linux operating systems. KodiBot is a standalone app and does not require an internet connection or additional dependencies to run local chat assistants. It supports both Llama.cpp compatible models and OpenAI API.
-
Mixture-of-Depths: Dynamically allocating compute in transformers
There are already some implementations out there which attempt to accomplish this!
Here's an example: https://github.com/silphendio/sliced_llama
A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587...
Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275
And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...
-
The lifecycle of a code AI completion
For those who might not be aware of this, there is also an open source project on GitHub called "Twinny" which is an offline Visual Studio Code plugin equivalent to Copilot: https://github.com/rjmacarthy/twinny
It can be used with a number of local model services. Currently for my setup on a NVIDIA 4090, I'm running both the base and instruct model for deepseek-coder 6.7b using 5_K_M Quantization GGUF files (for performance) through llama.cpp "server" where the base model is for completions and the instruct model for chat interactions.
llama.cpp: https://github.com/ggerganov/llama.cpp/
deepseek-coder 6.7b base GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...
deepseek-coder 6.7b instruct GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...
rust-gpu
-
Vcc – The Vulkan Clang Compiler
Sounds cool, but this requires yet another language to learn[0]. As someone who only has limited knowledge in this space, could someone tell me how comparable is the compute functionality of rust-gpu[1], where I can just write rust?
[0] https://github.com/Hugobros3/shady#language-syntax
[1] https://github.com/EmbarkStudios/rust-gpu
-
Candle: Torch Replacement in Rust
I don't do anything related to data science, but I feel like doing it in Rust would be nice.
You get operator overloading, so you can have ergonomic matrix operations that are typed also. Processing data on the CPU is fast, and crates like https://github.com/EmbarkStudios/rust-gpu make it very ergonomic to leverage the GPU.
I like this library for creating typed coordinate spaces for graphics programming (https://github.com/servo/euclid), I imagine something similar could be done to create refined types for matrices so you don't do matrix multiplication matrices of invalid sizes
-
What's the coolest Rust project you've seen that made you go, 'Wow, I didn't know Rust could do that!'?
Do you mean rust-gpu?
-
How a Nerdsnipe Led to a Fast Implementation of Game of Life
And https://github.com/EmbarkStudios/rust-gpu/tree/main/examples with the wgpu runner (here it runs the compute shader)
-
What is Rust's potential in game development?
I don't know how major they are considered, but Embark Studios is doing quite a bit of Rust in the open source space, most notably (IMO) rust-gpu and kajiya
-
[rust-gpu] How do I run/build my own shaders locally?
The examples in the rust-gpu repository are a good place to start
-
Posh: Type-Safe Graphics Programming in Rust
There's another project that's similar that's being used by an actual game company: https://github.com/EmbarkStudios/rust-gpu
They see specific advantages here that would outweigh that negative. It's not my space (I play games, but know next to nothing about graphics programming), but there's at least one argument in the other direction.
-
Introducing posh: Type-Safe Graphics Programming in Rust
Could this approach work for compute shaders (GPGPU) as well? So far, I think https://github.com/EmbarkStudios/rust-gpu is the state of the art in that area, but it adds a specific Rust compiler backend for generating SPIR-V rather than leaving that up to the driver. That seems more complicated than it needs to be... but maybe it has advantages too? Thoughts?
-
Looking for high level GPU computing crate
https://github.com/embarkstudios/rust-gpu Allows you to create shaders (kernals) in Rust.
-
With what languages are video games like League of Legends (most likely) programmed?
Also Embark Studios (formers DICE people) is doing a lot of work with Rust, all open source like Rust GPU https://github.com/EmbarkStudios/rust-gpu
What are some alternatives?
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
wgpu - Cross-platform, safe, pure-rust graphics api.
gpt4all - gpt4all: run open-source LLMs anywhere
Rust-CUDA - Ecosystem of libraries and tools for writing and executing fast GPU code fully in Rust.
text-generation-webui - A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.
onnxruntime-rs - Rust wrapper for Microsoft's ONNX Runtime (version 1.8)
GPTQ-for-LLaMa - 4 bits quantization of LLaMA using GPTQ
kompute - General purpose GPU compute framework built on Vulkan to support 1000s of cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.
ggml - Tensor library for machine learning
DiligentEngine - A modern cross-platform low-level graphics library and rendering framework
alpaca.cpp - Locally run an Instruction-Tuned Chat-Style LLM
compute-shader-101 - Sample code for compute shader 101 training