shumai VS tinygrad

https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-...

([...], tflite-support, tflite-micro)

From facebookresearch/shumai (a JS tensor library) https://github.com/facebookresearch/shumai/issues/122 :

> It doesn't make sense to support anything besides WebGPU at this point. WASM + SIMD is around 15-20x slower on my machine[1]. Although WebGL is more widely supported today, it doesn't have the compute features needed for efficient modern ML (transformers etc) and will likely be a deprecated backend for other frameworks when WebGPU comes online.

tensorflow rust has a struct.Tensor:

JS for ML actually does exist https://github.com/facebookresearch/shumai

> It's really a shame that data science, ML, and notebooks are so wrapped up in it. Otherwise we could jettison the whole thing into space

Although I personally feel Python has its place, I contribute to a project that hopes to diversify the ML/scientific computing space with a TypeScript tensor lib called Shumai: https://github.com/facebookresearch/shumai

Doesn’t really matter for large batch/large model training on GPUs that don’t need much coordination.

But Python speed is one of the main motivations for a JS/TS based ML lib I’m working on: https://github.com/facebookresearch/shumai

As a hedge against CPython never becoming fast, we're creating a project called Shumai that attempts to deeply integrate with a new JavaScript runtime (Bun[3]).

You can expose objects. Here's how it is done in Bun: https://github.com/facebookresearch/shumai/blob/main/shumai/...

We've been using this feature heavily in Shumai.

I think you are vastly overestimating the complexity associated with this (user exposed ref-counting/garbage collection) and may not be totally up to date on what's implemented.

This thought is pretty much the exact motivation behind a recent effort I’m helping out with https://github.com/facebookresearch/shumai

Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]?

[0] https://github.com/geohot/tinygrad

at first glance i thought may its like tinygrad. but looks has many ops than that tiny grad but most maps to underlying hardware provided ops?

i wonder how well tinygrad's apporach will work out, ops fusion sounds easy, just a walk a graph, pattern match it and lower to hardware provided ops?

Anyway if anyone wants to understand the philosophy behind tinygrad, this file is great start https://github.com/geohot/tinygrad/blob/master/docs/abstract...

There are currently at least 3 ways to run llama on m1 with GPU acceleration. - mlc-llm (pre-built, only 1 model has been ported) - tinygrad (very memory efficient, not that easy to integrate into other projects) - llama-mps (original llama codebase + llama adapter support)

Hopefully we will see now full support with AMD hardware on https://github.com/geohot/tinygrad. You can read more about it on https://tinygrad.org/

tinygrad

https://github.com/geohot/tinygrad/tree/master/accel/ane

But I have not tested it on Linux since Asahi has not yet added support.

llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) without quantization.