metal-cpp
nitro
Our great sponsors
metal-cpp | nitro | |
---|---|---|
16 | 7 | |
250 | 1,572 | |
- | 19.1% | |
3.3 | 9.8 | |
4 months ago | 5 days ago | |
C++ | C++ | |
Apache License 2.0 | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
metal-cpp
-
Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API
My understanding is the proliferation of “XYZ-cpp” AI frameworks is due to the c++ support in Apple’s gpu library ‘Metal’, and the popularity of apple silicon for inference (and there are a few technical reasons for this): https://developer.apple.com/metal/cpp/
-
Show HN: C-ocoa, Write iOS/macOS apps in any language, with a generated C API
This is basically also what the "official" C++ API for Metal does (https://developer.apple.com/metal/cpp/), it's an automatically generated bindings wrapper which calls into ObjC runtime functions.
I also dabbled a bit with this idea by parsing clang AST-dumps of macOS system headers:
https://github.com/floooh/objc-ast-experiments
Unfortunately this is very brittle, and also broke on ARM CPUs, I guess the shim code needs some ABI adjustments (famously, objc_msgSend has multiple "ABI shapes": https://www.mikeash.com/pyblog/objc_msgsends-new-prototype.h...).
-
What's the best way to learn Metal?
There's official C++-interface: https://developer.apple.com/metal/cpp/
- What are some alternatives to OpenGL for Mac
- Opinion for graphic api's?
-
A brief interview with Tcl creator John Ousterhout
It doesn't matter if the project driven by Microsoft or not, the cat (of automatically generated language bindings) is out of the bag. E.g. Zig is using the same approach without being an official MS project: https://github.com/marlersoft/zigwin32, and Apple has an automatically generated C++ API for Metal (https://developer.apple.com/metal/cpp/).
In the future, the question won't be "what language do I need to learn to code on this platform", but instead "are there language bindings for my favourite language".
- Cross platform low level graphics API suitable for game development?
-
GCC now includes Modula-2 and Rust. Do they work on OpenBSD?
this? https://developer.apple.com/metal/cpp/
Doesn't it just use objc/runtime.h and if anything is missing you can just add your custom api calls?
-
A learning path for Vulkan that focuses on concepts?
Metal has C++ bindings (which cover a full app lifecycle so you don’t have to touch Objective-C/Swift at all) but they’re based on the Objective-C memory model. There are some helper structs mimicking shared pointers, but you’ll still need to understand the basics of how an autorelease pool is used to avoid memory leaks and/or bad access crashes.
-
CTO of Azure declares C++ "deprecated"
On https://developer.apple.com/metal/cpp/ check Foundation folder and all those nice Object::sendMessage().
nitro
-
Ollama Python and JavaScript Libraries
I'd like to see a comparison to nitro https://github.com/janhq/nitro which has been fantastic for running a local LLM.
- FLaNK Weekly 08 Jan 2024
-
Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API
Look... I appreciate a cool project, but this is probably not a good idea.
> Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.
It's not. It's literally just llama.cpp -> https://github.com/janhq/nitro/blob/main/.gitmodules
Llama.cpp makes no pretense at being a robust safe network ready library; it's a high performance library.
You've made no changes to llama.cpp here; you're just calling the llama.cpp API directly from your drogon app.
Hm.
...
Look... that's interesting, but, honestly, I know there's this wave of "C++ is back!" stuff going on, but building network applications in C++ is very tricky to do right, and while this is cool, I'm not sure 'llama.cpp is in c++ because it needs to be fast' is a good reason to go 'so lets build a network server in c++ too!'.
I mean, I guess you could argue that since llama.cpp is a C++ application, it's fair for them to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216, https://github.com/ggerganov/llama.cpp/blob/master/examples/...).
...but a production ready application?
I wrote a rust binding to llama.cpp and my conclusion was that llama.cpp is pretty bleeding edge software, and bluntly, you should process isolate it from anything you really care about, if you want to avoid undefined behavior after long running inference sequences; because it updates very often, and often breaks. Those breaks are usually UB. It does not have a 'stable' version.
Further more, when you run large models and run out of memory, C++ applications are notoriously unreliable in their 'handle OOM' behaviour.
Soo.... I know there's something fun here, but really... unless you had a really really compelling reason to need to write your server software in c++ (and I see no compelling reason here), I'm curious why you would?
It seems enormously risky.
The quality of this code is 'fun', not 'production ready'.
- Apple Silicon Llama 7B running in docker?
- Is there any LLM that can be installed with out python
What are some alternatives?
MoltenVK - MoltenVK is a Vulkan Portability implementation. It layers a subset of the high-performance, industry-standard Vulkan graphics and compute API over Apple's Metal graphics framework, enabling Vulkan applications to run on macOS, iOS and tvOS.
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
Cinder - Cinder is a community-developed, free and open source library for professional-quality creative coding in C++.
bionic-gpt - BionicGPT is an on-premise replacement for ChatGPT, offering the advantages of Generative AI while maintaining strict data confidentiality
LearnOpenGL - Code repository of all OpenGL chapters from the book and its accompanying website https://learnopengl.com
nnl - a low-latency and high-performance inference engine for large models on low-memory GPU platform.
objc4
csvlens - Command line csv viewer
metal-rs - Rust bindings for Metal
Tribuo - Tribuo - A Java machine learning library
OpenFrameworks - openFrameworks is a community-developed cross platform toolkit for creative coding in C++.
hyperfine - A command-line benchmarking tool