Petalisp VS criterium

While not "as fast as C" (C is not the absolute pinnacle of performance), Common Lisp is incredibly fast compared to the majority of programming languages around today. There is even a huge amount of ongoing work being done to make it faster still. We are seeing many interesting projects that make better use of the hardware in your computer (e.g. https://github.com/marcoheisig/Petalisp).

i think lisp-stat library is actually being developed. however one numerical cl library that doesnt get enough mention and is being constantly developed is petalisp for HPC

https://github.com/marcoheisig/Petalisp

However, if you have a lisp library that puts those semantics to use, then you could get it to employ magicl/ext-blas and cl-bmas to speed it up. (petalisp looks relevant, but I lack the background to compare it with APL.)

> his means cl pagckages can be "done".

this is true if there is nothing functional that can be added to a package. however its very much not true for ml frameworks right now. new things are being added all the time in the field. however even in the package i linked you have the necessary ingredients for any deep learning model: cuda and back propagation. the other person mentioned convolution which i think is pretty trivial to implement but still, if you expect everything for you to be ready made then you should probably stick to tf and pytorch. if you want to explore the cutting edge and push the boundaries then i think common lisp is a good tool. as an aside it might also be interesting to note that a common lisp package (Petalisp) is being used for high performance computing by a german university

https://github.com/marcoheisig/Petalisp

One trick is that Common Lisp can generate and compile code at runtime, whereas static languages typically do not have a compiler available at runtime. This lets you make your own lazy person's JIT/staged compiler, which is useful if some part of the problem is not known at compile-time. Such an approach has been used at least for array munging, type munging and regular expression munging.

(criterium does not work here yet b.t.w., but it probably will be working soon)

criterium, specifically the quick-bench function, will actually run multiple samples an provide a mean runtime (as well as other useful stats) so you can get an idea of what a jit'd warmed up performance looks like. time is great in a pinch, but you end up needing to run it multiple times to ensure optimizations are kicking and and other artifacts (like gc) aren't throwing the results.

I'm going to leave tooling out of this and run everything through a repl on the command line right from the jar. One other thing I want to do is include the incredible criterium library so we can profile. I'm deliberately including criterium separately like this because you shouldn't have a dev-time tool like criterium in an uberjar. And knowing how to easily combine other jars with your real production jar can be very helpful. I grabbed the jar from my .m2 cache.

I am just going to leave this here - https://github.com/hugoduncan/criterium

where crit is criterium. As you can see, you're spending most of your time in the seq transformation part.

It's better to benchmark with something like criterium. time is a bit inaccurate. Though, if it's really 15 seconds, I guess will not be that big of a difference

>>> One of Clojure's biggest weaknesses in practice is that breaking in to those functional structures to figure out where the time is being spent or to debug them is harder than in other languages. This is a natural trade-off of developing a terse and powerful language.

Not that hard if you use something like YourKit. There's also a quite good Clojure library https://github.com/hugoduncan/criterium .

Criterium (the benchmarking library used here) uses multiple runs to obtain tighter bounds on amortized performance, as well as techniques to amortize the effects of garbage collection and JIT compilation. See https://github.com/hugoduncan/criterium for a brief overview, as well as links to the pitfalls and statistical techniques involved in JVM benchmarking.