TornadoVM VS stable_diffusion.openvino

You don't need to use C++ to interface with CUDA or even write it.

A while ago NVIDIA and the GraalVM team demoed grCUDA which makes it easy to share memory with CUDA kernels and invoke them from any managed language that runs on GraalVM (which includes JIT compiled Python). Because it's integrated with the compiler the invocation overhead is low:

https://developer.nvidia.com/blog/grcuda-a-polyglot-language...

And TornadoVM lets you write kernels in JVM langs that are compiled through to CUDA:

https://www.tornadovm.org

There are similar technologies for other languages/runtimes too. So I don't think that will cause NVIDIA to lose ground.

Presented by Juan Fumero, PhD & Research Fellow (The University of Manchester, UK) during the JVM Language Summit 2023 (Santa Clara CA).

More information on TornadoVM can be found at https://www.tornadovm.org/

Tags: #Java #JVMLS #GPU #FPGA #OpenJDK #GraalVM #AI

it might be the --xformers flag, try getting rid of that since your not using cuda you wouldn't be able to run it with xformers and you could also try --use-cpu all ... you can also check this out .. https://github.com/bes-dev/stable_diffusion.openvino .. it's probably your best option if your using CPU, which if your PC Graphics are using Intel UHD 620 then you don't have a GPU and an optimized CPU inference would be best to run

SD can be run on a CPU without a GPU. I know for certain it can be done with OpenVINO. In fact, on some i7s, it will run at around 3 seconds per iteration. There was a reddit SD thread a while back saying it can be done with Automatic111. Also, soe recent threads on problems with AMD GPUs suggest Automatic1111 is using the CPU rather than the intended GPU. (Fortuanely, I have a GPU, so I don't have to deal with it myself!)

I'm not actually entirely convinced that it's even using the GPU. Radeontop shows 0% utilization while the images are generating. Additionally, the listed iteration speed should be impossibly slow for any GPU; it says 26.58s/it, which is slower than just running on a CPU.

iGPU's are in short not supported. There's this repo that may or may not help you, but even if it did I wouldn't expect much.

You can also run it in windows native with openvino, there is a barebones webui for it as well in one of the forks.Requires setting cpu to gpu in one the files. https://github.com/bes-dev/stable_diffusion.openvino

In https://github.com/bes-dev/stable_diffusion.openvino/blob/master/stable_diffusion_engine.py

" We ended up using three different Stable Diffusion projects for our testing, mostly because no single package worked on every GPU. For Nvidia, we opted for Automatic 1111's webui version(opens in new tab). AMD GPUs were tested using Nod.ai's Shark version(opens in new tab), while for Intel's Arc GPUs we used Stable Diffusion OpenVINO(opens in new tab). "