numcl VS Petalisp

Is it me or numcl is faster than magicl? Matrix multiplication on magicl with pure lisp backend is

Interesting, I will, thanks! I am aware of numcl for CL, but I don't think it is "there" yet :).

The first that comes to mind is numcl. This works if (i) performance is not seriously a concern, (ii) you are not annoyed by julia-like JIT/JAOT compilation delays, (iii) copy-based slicing won't be a performance issue for you. To be fair, limitation (i) might be overcome by writing a better (simd-based) backend for numcl. numcl is fast, it compiles to fairly good code, but simd can boost the performance by another 4-8 times or so.

am I missing something that already exists (numcl / Alexandria / core language, etc?)

>Either implement numpy equivalent on your own or half of your code is data massaging data between libraries

I haven't tested this but here you go:

https://github.com/numcl/numcl

Personally, I've been relying on the stream-based method using py4cl/2, mostly because I did not - and perhaps do not - have the knowledge and time to dig into the CFFI based method. The limitation is that this would get you less than 10000 python interactions per second. That is sufficient if you will be running a long running python task - and I have successfully run trivial ML programs using it, but any intensive array processing gets in the way. For this later task, there are a few emerging libraries like numcl and array-operations without SIMD (yet), and numericals using SIMD. For reasons mentioned on the readme, I recently cooked up dense-arrays. This has interchangeable backends and can also use cl-cuda. But barring that, the developer overhead of actually setting up native-CFFI ecosystem is still too high, and I'm back to py4cl/2 for tasks beyond array processing.

numcl - Numpy clone in Common Lisp. [LGPL3][9].

[3] https://github.com/numcl/numcl

While not "as fast as C" (C is not the absolute pinnacle of performance), Common Lisp is incredibly fast compared to the majority of programming languages around today. There is even a huge amount of ongoing work being done to make it faster still. We are seeing many interesting projects that make better use of the hardware in your computer (e.g. https://github.com/marcoheisig/Petalisp).

i think lisp-stat library is actually being developed. however one numerical cl library that doesnt get enough mention and is being constantly developed is petalisp for HPC

https://github.com/marcoheisig/Petalisp

However, if you have a lisp library that puts those semantics to use, then you could get it to employ magicl/ext-blas and cl-bmas to speed it up. (petalisp looks relevant, but I lack the background to compare it with APL.)

> his means cl pagckages can be "done".

this is true if there is nothing functional that can be added to a package. however its very much not true for ml frameworks right now. new things are being added all the time in the field. however even in the package i linked you have the necessary ingredients for any deep learning model: cuda and back propagation. the other person mentioned convolution which i think is pretty trivial to implement but still, if you expect everything for you to be ready made then you should probably stick to tf and pytorch. if you want to explore the cutting edge and push the boundaries then i think common lisp is a good tool. as an aside it might also be interesting to note that a common lisp package (Petalisp) is being used for high performance computing by a german university

https://github.com/marcoheisig/Petalisp

One trick is that Common Lisp can generate and compile code at runtime, whereas static languages typically do not have a compiler available at runtime. This lets you make your own lazy person's JIT/staged compiler, which is useful if some part of the problem is not known at compile-time. Such an approach has been used at least for array munging, type munging and regular expression munging.