CLBlast VS ArrayFire

git clone https://github.com/CNugteren/CLBlast.git cd CLBlast cmake . cmake --build . --config Release mkdir install cmake --install . --prefix ~/CLBlast/install cp libclblast.so* $PREFIX/lib cp ./include/clblast.h ../llama.cpp

I'm trying to get GPU-Acceleration to work with oobabooga's webui, there it says that I just have to reinstall the llama-cpp-python in the environment and have it compile with CLBLAST.So I have CLBLAST downloaded and unzipped, but when I try to do it with:

Which raspberry Pi? For Pi 1 - 3, you can use VC4CL . While it's an impressive effort, it is highly experimental and does not always work as it should. I spent some non-trivial time trying to get CLBlast (a BLAS implementation for OpenCL) working on a 3b+, but there's always something hanging or not giving the right results.

Install CLBlast: cd git clone https://github.com/CNugteren/CLBlast.git cd CLBlast cmake -B build \ -DBUILD_SHARED_LIBS=OFF \ -DTUNERS=OFF \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/data/data/com.termux/files/usr cd build make -j8 make install

Loads of people have stated why easy GPU interfaces are difficult to create, but we solve many difficult things all the time.

Ultimately I think CPUs are just satisfactory for the vast vast majority of workloads. Servers rarely come with any GPUs to speak of. The ecosystem around GPUs is unattractive. CPUs have SIMD instructions that can help. There are so many reasons not to use GPUs. By the time anyone seriously considers using GPUs they're, in my imagination, typically seriously starved for performance, and looking to control as much of the execution details as possible. GPU programmers don't want an automagic solution.

So I think the demand for easy GPU interfaces is just very weak, and therefore no effort has taken off. The amount of work needed to make it as easy to use as CPUs is massive, and the only reason anyone would even attempt to take this on is to lock you in to expensive hardware (see CUDA).

For a practical suggestion, have you taken a look at https://arrayfire.com/ ? It can run on both CUDA and OpenCL, and it has C++, Rust and Python bindings.

What about Arrayfire. https://github.com/arrayfire/arrayfire

Low-overhead — not our goal, but Flashlight is on par with or outperforming most other ML/DL frameworks with its ArrayFire reference tensor implementation, especially on nonstandard setups where framework overhead matters

Looking for frameworks with Julia + OpenCL I found array fire. It seems quite good, bonus points for rust bindings. I will keep looking for more, Julia completely fell off my radar.

Your Problem may be the lazy evaluation, see this issue: https://github.com/arrayfire/arrayfire/issues/1709