py4cl2 VS numcl

I... recently got that working: https://github.com/digikar99/py4cl2/tree/master/cffi - Yes, CFFI! Yes, passing CL array data by reference!

I ended up using a fair bit of matplotlib through college and with colleagues. I too don't want to use python, but I also don't like throwing away its libraries, and I'm too lazy to invest in other* plotting ecosystems. In effect, I use up using matplotlib through py4cl/2.

Note that it is not my aim to replace the python ecosystem; I think that is far too lofy a goal to be of any good. My original intention was to interoperate with python through py4cl/2 or the likes, but felt that one needs a Common Lisp library for "small" operations, while "large" operations can be offloaded to python libraries through py4cl/2.

Python: Blender and Panda3D (game engine used for Disney's Toontown way back when) are both scriptable with Python. I've been able to successfully call Panda from Py4CL2 (thanks digikar for the help with that), but I have not tried with Blender yet. I think it's doable.

While you are at it I may as well mention https://github.com/digikar99/py4cl2

If performance is absolutely not a concern, then third option is using python libraries through py4cl/2. To put it differently, if calling python from lisp is not the bottleneck, then this is a feasible option.

py4cl2 (not py4cl!) author here. From the v2.9.0 docs:

py4cl and py4cl2 represent a fairly pragmatic example of method 1, using an OS child process to communicate back and forth with your python code. Python is fairly popular and well-enabled with libraries, so you can delegate things to python that leverage those libraries.

If you can structure your code so that data de/serialization is not a bottleneck, then you could access the python libraries using py4cl/2.

Is it me or numcl is faster than magicl? Matrix multiplication on magicl with pure lisp backend is

Interesting, I will, thanks! I am aware of numcl for CL, but I don't think it is "there" yet :).

The first that comes to mind is numcl. This works if (i) performance is not seriously a concern, (ii) you are not annoyed by julia-like JIT/JAOT compilation delays, (iii) copy-based slicing won't be a performance issue for you. To be fair, limitation (i) might be overcome by writing a better (simd-based) backend for numcl. numcl is fast, it compiles to fairly good code, but simd can boost the performance by another 4-8 times or so.

am I missing something that already exists (numcl / Alexandria / core language, etc?)

>Either implement numpy equivalent on your own or half of your code is data massaging data between libraries

I haven't tested this but here you go:

https://github.com/numcl/numcl

Personally, I've been relying on the stream-based method using py4cl/2, mostly because I did not - and perhaps do not - have the knowledge and time to dig into the CFFI based method. The limitation is that this would get you less than 10000 python interactions per second. That is sufficient if you will be running a long running python task - and I have successfully run trivial ML programs using it, but any intensive array processing gets in the way. For this later task, there are a few emerging libraries like numcl and array-operations without SIMD (yet), and numericals using SIMD. For reasons mentioned on the readme, I recently cooked up dense-arrays. This has interchangeable backends and can also use cl-cuda. But barring that, the developer overhead of actually setting up native-CFFI ecosystem is still too high, and I'm back to py4cl/2 for tasks beyond array processing.

numcl - Numpy clone in Common Lisp. [LGPL3][9].

[3] https://github.com/numcl/numcl