burn VS llama2.rs

View the Project on GitHub

I intend to grow this into an open-source project because deep inside, this is ideally how I would like ComfyUI to be. There's still a long journey ahead for building all the custom nodes, which is especially challenging given that the majority of code for AI workflows is written in Python. However, with my hands-on experience with Candle and Burn libraries, I may be able to get pretty close!

The need to build CubeCL came from the Burn deep learning framework (https://github.com/tracel-ai/burn), where we want to easily build algorithms like in CUDA with a real programming language, while also being able to integrate those algorithms inside a compiler at runtime to fuse dynamic graphs.

Since we don't want to rewrite everything multiple times, it also has to be multi-platform and optimal, so the feature set must be per-device, not per-language. I'm not aware of a tool that does that, especially in Rust (which Burn is written in).

Burn is an emerging deep learning framework written in pure Rust that aims to provide a flexible, efficient, and safe environment for building and training neural networks. With its modular design and strong type system, Burn represents a significant step forward in bringing deep learning to the Rust ecosystem.

7. Burn Burn is a dynamic deep-learning framework built with flexibility and efficiency in mind. If you're into AI or machine learning, this framework offers the ability to explore how Rust can power complex neural networks.

You can use libtorch directly via `tch-rs`, and at present I'm porting over to Burn (see https://burn.dev) which appears incredibly promising. My impression is it's in a good place, if of course not close to the ecosystem of Python/C++. At very least I've gotten my nn models training and running without too much difficulty. (I'm moving to Burn for the thread safety - their `Tensor` impl is `Sync` - libtorch doesn't have such a guarantee.)

Burn has Candle as one of its backends, which I understand is also quite popular.

[package] name = "resnet_burn" version = "0.1.0" edition = "2021" [dependencies] burn = { git = "https://github.com/tracel-ai/burn.git", rev = "75cb5b6d5633c1c6092cf5046419da75e7f74b11", features = ["ndarray"] } burn-import = { git = "https://github.com/tracel-ai/burn.git", rev = "75cb5b6d5633c1c6092cf5046419da75e7f74b11" } image = { version = "0.24.7", features = ["png", "jpeg"] }

This code runs Llama2 quantized and unquantized in a roughly minimal way: https://github.com/srush/llama2.rs (though extracting the quantized 70B weights takes a lot of RAM). I'm running the 13B quantized model on ~10-11GB of CPU memory.

Nowhere near as neat as candle or ggml, but just released a 4-bit rust llama2 implementation with simd. Runs pretty fast.

https://github.com/srush/llama2.rs/