py4cl2 VS cl-cuda

I... recently got that working: https://github.com/digikar99/py4cl2/tree/master/cffi - Yes, CFFI! Yes, passing CL array data by reference!

I ended up using a fair bit of matplotlib through college and with colleagues. I too don't want to use python, but I also don't like throwing away its libraries, and I'm too lazy to invest in other* plotting ecosystems. In effect, I use up using matplotlib through py4cl/2.

Note that it is not my aim to replace the python ecosystem; I think that is far too lofy a goal to be of any good. My original intention was to interoperate with python through py4cl/2 or the likes, but felt that one needs a Common Lisp library for "small" operations, while "large" operations can be offloaded to python libraries through py4cl/2.

Python: Blender and Panda3D (game engine used for Disney's Toontown way back when) are both scriptable with Python. I've been able to successfully call Panda from Py4CL2 (thanks digikar for the help with that), but I have not tried with Blender yet. I think it's doable.

While you are at it I may as well mention https://github.com/digikar99/py4cl2

If performance is absolutely not a concern, then third option is using python libraries through py4cl/2. To put it differently, if calling python from lisp is not the bottleneck, then this is a feasible option.

py4cl2 (not py4cl!) author here. From the v2.9.0 docs:

py4cl and py4cl2 represent a fairly pragmatic example of method 1, using an OS child process to communicate back and forth with your python code. Python is fairly popular and well-enabled with libraries, so you can delegate things to python that leverage those libraries.

If you can structure your code so that data de/serialization is not a bottleneck, then you could access the python libraries using py4cl/2.

> You can write a lot of macrology to get around it, but there's a point where you want actual compiler writers to be doing this

this is not the job of compiler writers (although writing macros is akin to writing a compiler but i do not think that this is what you mean). in julia the numerical programming packages are not part of the standard library and a lot of it is wrappers around C++ code especially when the drivers to the underlining hardware are closed-source [0]. also here is the similar library in common lisp [1]

[0] https://github.com/JuliaGPU/CUDA.jl

[1] https://github.com/takagi/cl-cuda

A Common Lisp Library to Use Nvidia CUDA\ (0 comments)

Personally, I've been relying on the stream-based method using py4cl/2, mostly because I did not - and perhaps do not - have the knowledge and time to dig into the CFFI based method. The limitation is that this would get you less than 10000 python interactions per second. That is sufficient if you will be running a long running python task - and I have successfully run trivial ML programs using it, but any intensive array processing gets in the way. For this later task, there are a few emerging libraries like numcl and array-operations without SIMD (yet), and numericals using SIMD. For reasons mentioned on the readme, I recently cooked up dense-arrays. This has interchangeable backends and can also use cl-cuda. But barring that, the developer overhead of actually setting up native-CFFI ecosystem is still too high, and I'm back to py4cl/2 for tasks beyond array processing.