Petalisp VS magicl

While not "as fast as C" (C is not the absolute pinnacle of performance), Common Lisp is incredibly fast compared to the majority of programming languages around today. There is even a huge amount of ongoing work being done to make it faster still. We are seeing many interesting projects that make better use of the hardware in your computer (e.g. https://github.com/marcoheisig/Petalisp).

i think lisp-stat library is actually being developed. however one numerical cl library that doesnt get enough mention and is being constantly developed is petalisp for HPC

https://github.com/marcoheisig/Petalisp

However, if you have a lisp library that puts those semantics to use, then you could get it to employ magicl/ext-blas and cl-bmas to speed it up. (petalisp looks relevant, but I lack the background to compare it with APL.)

> his means cl pagckages can be "done".

this is true if there is nothing functional that can be added to a package. however its very much not true for ml frameworks right now. new things are being added all the time in the field. however even in the package i linked you have the necessary ingredients for any deep learning model: cuda and back propagation. the other person mentioned convolution which i think is pretty trivial to implement but still, if you expect everything for you to be ready made then you should probably stick to tf and pytorch. if you want to explore the cutting edge and push the boundaries then i think common lisp is a good tool. as an aside it might also be interesting to note that a common lisp package (Petalisp) is being used for high performance computing by a german university

https://github.com/marcoheisig/Petalisp

One trick is that Common Lisp can generate and compile code at runtime, whereas static languages typically do not have a compiler available at runtime. This lets you make your own lazy person's JIT/staged compiler, which is useful if some part of the problem is not known at compile-time. Such an approach has been used at least for array munging, type munging and regular expression munging.

(Link didn't work for me)

https://github.com/quil-lang/magicl/blob/master/src/high-lev...

use MAGICL. [1] It is optionally and transparently accelerated by BLAS/LAPACK.

[1] https://github.com/quil-lang/magicl/blob/master/doc/high-lev...

Maybe have a look at how magicl does this?

This is essentially the first option, which is already supported by MAGICL by loading MAGICL/EXT-LAPACK [1].

[1] https://github.com/quil-lang/magicl#extensions

I think what the magicl team has done is brilliant - allowing multiple implementations is awesome.

Second up is magicl, especially useful if performance is a concern. This might not be as extensive as numcl, but it's been battle tested in the industry over the last decade or so. Because this uses generic functions, so long as you are using not-very-small arrays, performance should not be a concern for you. And even if you are, you could write your own functions that use the low-level functions that magicl's backends define. Otherwise performance can be at par with numpy.

This Dev How-To describes (I hope in enough detail) how to add these specialized routines to MAGICL.

I agree... and I do don't want be the person who has not rallied. I just took a look at guicho's issue from 2019. And here, you yourself have admitted that the high level interface is less than ideal and needs more work. However, the very point that magicl is an industry standard could imply that potentially radical backward-incompatible changes can be hard. But, honestly, I want to discuss this, time permitting!

Common Lisp is a great language to build new tools for data science, but currently has pretty awful library support existing data science workflows. Common Lisp is sorely lacking in high-quality statistics, plotting, and sparse arrays. There’s been a long work-in-progress library to bring flexible and high-performance linear algebra to Lisp, but it needs more contributors.