magicl VS neanderthal

(Link didn't work for me)

https://github.com/quil-lang/magicl/blob/master/src/high-lev...

use MAGICL. [1] It is optionally and transparently accelerated by BLAS/LAPACK.

[1] https://github.com/quil-lang/magicl/blob/master/doc/high-lev...

Maybe have a look at how magicl does this?

This is essentially the first option, which is already supported by MAGICL by loading MAGICL/EXT-LAPACK [1].

[1] https://github.com/quil-lang/magicl#extensions

I think what the magicl team has done is brilliant - allowing multiple implementations is awesome.

Second up is magicl, especially useful if performance is a concern. This might not be as extensive as numcl, but it's been battle tested in the industry over the last decade or so. Because this uses generic functions, so long as you are using not-very-small arrays, performance should not be a concern for you. And even if you are, you could write your own functions that use the low-level functions that magicl's backends define. Otherwise performance can be at par with numpy.

This Dev How-To describes (I hope in enough detail) how to add these specialized routines to MAGICL.

I agree... and I do don't want be the person who has not rallied. I just took a look at guicho's issue from 2019. And here, you yourself have admitted that the high level interface is less than ideal and needs more work. However, the very point that magicl is an industry standard could imply that potentially radical backward-incompatible changes can be hard. But, honestly, I want to discuss this, time permitting!

Common Lisp is a great language to build new tools for data science, but currently has pretty awful library support existing data science workflows. Common Lisp is sorely lacking in high-quality statistics, plotting, and sparse arrays. There’s been a long work-in-progress library to bring flexible and high-performance linear algebra to Lisp, but it needs more contributors.

There is the official Hello World https://github.com/uncomplicate/neanderthal/tree/master/examples/hello-world

Did you have any occasion to evaluate neanderthal during your research? People seem to prefer it over core.matrix because it focus on primitive speed and sticking to BLAS idioms (as well as offering a decent api for working with GPU backends via cuda and opencl). I am curious to see if you did and found anything lacking there. I have a project on the backburner to try and target neanderthal for local search stuff, expressing problems in a high-level API that can then be baked into some numerically-friendly representation for efficient execution. It's often easier (trivial) to express solution representations, neighborhood functions, and objectives/constraints in a general purpose language, of which none of the things we like (sparse data structures, dynamically allocated stuff) are amenable to the contiguous memory, primitive numeric model that the hardware wants.

Do clojure as a side gig or in free time. Let day job pay the bills. If you can, maybe incorporate clojure into work job to solve small problems (https://github.com/clj-python/libpython-clj and https://github.com/scicloj/clojisr provide bridges to/from python and r). There is a lot of effort going into the data science side as well; the scicloj effort has resulted in a lot of growth over the last 2 years. tech.ml.dataset, tech.ml (now scicloj.ml). Dragan has a bunch of excellent stuff in neanderthal and deep diamond. There are also bindings to other jvm libraries from multiple languages.