metal-cpp
llama.cpp
Our great sponsors
metal-cpp | llama.cpp | |
---|---|---|
16 | 769 | |
250 | 56,891 | |
- | - | |
3.3 | 10.0 | |
4 months ago | about 2 hours ago | |
C++ | C++ | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
metal-cpp
-
Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API
My understanding is the proliferation of “XYZ-cpp” AI frameworks is due to the c++ support in Apple’s gpu library ‘Metal’, and the popularity of apple silicon for inference (and there are a few technical reasons for this): https://developer.apple.com/metal/cpp/
-
Show HN: C-ocoa, Write iOS/macOS apps in any language, with a generated C API
This is basically also what the "official" C++ API for Metal does (https://developer.apple.com/metal/cpp/), it's an automatically generated bindings wrapper which calls into ObjC runtime functions.
I also dabbled a bit with this idea by parsing clang AST-dumps of macOS system headers:
https://github.com/floooh/objc-ast-experiments
Unfortunately this is very brittle, and also broke on ARM CPUs, I guess the shim code needs some ABI adjustments (famously, objc_msgSend has multiple "ABI shapes": https://www.mikeash.com/pyblog/objc_msgsends-new-prototype.h...).
-
What's the best way to learn Metal?
There's official C++-interface: https://developer.apple.com/metal/cpp/
- What are some alternatives to OpenGL for Mac
- Opinion for graphic api's?
-
A brief interview with Tcl creator John Ousterhout
It doesn't matter if the project driven by Microsoft or not, the cat (of automatically generated language bindings) is out of the bag. E.g. Zig is using the same approach without being an official MS project: https://github.com/marlersoft/zigwin32, and Apple has an automatically generated C++ API for Metal (https://developer.apple.com/metal/cpp/).
In the future, the question won't be "what language do I need to learn to code on this platform", but instead "are there language bindings for my favourite language".
- Cross platform low level graphics API suitable for game development?
-
GCC now includes Modula-2 and Rust. Do they work on OpenBSD?
this? https://developer.apple.com/metal/cpp/
Doesn't it just use objc/runtime.h and if anything is missing you can just add your custom api calls?
-
A learning path for Vulkan that focuses on concepts?
Metal has C++ bindings (which cover a full app lifecycle so you don’t have to touch Objective-C/Swift at all) but they’re based on the Objective-C memory model. There are some helper structs mimicking shared pointers, but you’ll still need to understand the basics of how an autorelease pool is used to avoid memory leaks and/or bad access crashes.
-
CTO of Azure declares C++ "deprecated"
On https://developer.apple.com/metal/cpp/ check Foundation folder and all those nice Object::sendMessage().
llama.cpp
-
Phi-3 Weights Released
well https://github.com/ggerganov/llama.cpp/issues/6849
- Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding
- Llama.cpp Working on Support for Llama3
-
Embeddings are a good starting point for the AI curious app developer
Have just done this recently for local chat with pdf feature in https://recurse.chat. (It's a macOS app that has built-in llama.cpp server and local vector database)
Running an embedding server locally is pretty straightforward:
- Get llama.cpp release binary: https://github.com/ggerganov/llama.cpp/releases
- Mixtral 8x22B
- Llama.cpp: Improve CPU prompt eval speed
-
Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
Ah, thanks for this! I can't edit my parent comment that you replied to any longer unfortunately.
As I said, I only compared the contributors graphs [0] and checked for overlaps. But those apparently only go back about year and only list at most 100 contributors ranked by number of commits.
[0]: https://github.com/ollama/ollama/graphs/contributors and https://github.com/ggerganov/llama.cpp/graphs/contributors
-
KodiBot - Local Chatbot App for Desktop
KodiBot is a desktop app that enables users to run their own AI chat assistants locally and offline on Windows, Mac, and Linux operating systems. KodiBot is a standalone app and does not require an internet connection or additional dependencies to run local chat assistants. It supports both Llama.cpp compatible models and OpenAI API.
-
Mixture-of-Depths: Dynamically allocating compute in transformers
There are already some implementations out there which attempt to accomplish this!
Here's an example: https://github.com/silphendio/sliced_llama
A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587...
Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275
And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...
-
The lifecycle of a code AI completion
For those who might not be aware of this, there is also an open source project on GitHub called "Twinny" which is an offline Visual Studio Code plugin equivalent to Copilot: https://github.com/rjmacarthy/twinny
It can be used with a number of local model services. Currently for my setup on a NVIDIA 4090, I'm running both the base and instruct model for deepseek-coder 6.7b using 5_K_M Quantization GGUF files (for performance) through llama.cpp "server" where the base model is for completions and the instruct model for chat interactions.
llama.cpp: https://github.com/ggerganov/llama.cpp/
deepseek-coder 6.7b base GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...
deepseek-coder 6.7b instruct GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...
What are some alternatives?
MoltenVK - MoltenVK is a Vulkan Portability implementation. It layers a subset of the high-performance, industry-standard Vulkan graphics and compute API over Apple's Metal graphics framework, enabling Vulkan applications to run on macOS, iOS and tvOS.
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
Cinder - Cinder is a community-developed, free and open source library for professional-quality creative coding in C++.
gpt4all - gpt4all: run open-source LLMs anywhere
LearnOpenGL - Code repository of all OpenGL chapters from the book and its accompanying website https://learnopengl.com
text-generation-webui - A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.
objc4
GPTQ-for-LLaMa - 4 bits quantization of LLaMA using GPTQ
metal-rs - Rust bindings for Metal
ggml - Tensor library for machine learning
OpenFrameworks - openFrameworks is a community-developed cross platform toolkit for creative coding in C++.
alpaca.cpp - Locally run an Instruction-Tuned Chat-Style LLM