42nd-at-threadmill VS numericals

I agree that "C/C++" isn't a good sign, though it was more forgivable when C++ just meant C++98... For speed, nowadays, you use X. That is, many languages can be fast, especially if they have access to intrinsics, the real question is how much special knowledge and herculean effort do you have to spend? It's still true that for many problems idiomatic modern C++ will give a very nice result that may be hard to beat (though be careful if you forgot cin.sync_with_stdio(false) or you may be slower than Python!). But it's also true that C, Rust, Java, Common Lisp, and even JS all do very well out of the box these days, while languages like Python and Ruby have lagged. For this problem, the author had to spend quite a bit of effort in a followup post to get Rust to match some 20 year old unoptimized CL.

If one wanted to optimize Lisp, a simple place to start is with some type declarations (at least with SBCL). It can even be kind of fun to have the compiler yell at you because e.g. it couldn't infer a type somewhere and was forced to do a generic add, so you tell it some things are fixnums, see the message go away, and verify if you want with DISASSEMBLE that it's now using LEA instead of a CALL. For an example of going to quite a bit of trouble (relatively) with adding inline hints, removing unneeded array bounds checks, and SIMD instructions, see https://github.com/telekons/42nd-at-threadmill which I believe is quite a bit faster than a similar "blazing fast" Rust project featured on HN not long ago. But my point again isn't so much that CL is the fastest or has some fast projects, just that it can be made as fast as you're probably going to want. This applies to a lot of other languages too, though CL still feels pretty unique in supporting both high level and low level constructs.

Your GC comment is weird, since any publications looking at total performance invariable include the GC overhead, nothing is hidden. The TIME macro for micro-benchmarking even typically reports on allocated memory, which can be a proxy for estimating GC pressure, and both SBCL and CCL report the actual time (if any) spent in GC. Why not complain that C++ benchmarks hide the indirect costs of memory fragmentation, which is a real bane for long-running C++ programs like application servers? But I'll admit that the GC can be a big weakness and it's no fun to be forced to code around its limitations, and historically some GCs used by some Lisps were really bad (that is, huge pause times). I've been looking at the herculean GC work being done in the Java ecosystem for years with a jealous eye, and even the newer Nim with its swappable GCs when you want to make certain tradeoffs without having to code them in.

Having beaten performance out of SBCL before, it seems...unlikely that there would be a benefit to fixnums when everything is inline and unboxed, except for isqrt, but AIUI we only compute it once per sieve so I'm not terribly bothered by that.

How about the semantics? Nevermind, I looked -- utter nonsense, just like numpy.

Then there is a question - do you actually need these libraries? You can optimize code in Common Lisp (type declarations, usage of appropriate data structures, SIMD instructions etc). See this: https://github.com/digikar99/numericals/tree/master/sbcl-numericals <- SIMD instructions used from SBCL (on x86; these are processor-family specific so Apple M1 will have different ones).

*I have not; I have a couple of WIP/alpha-stage libraries like dense-arrays and numericals that could be useful; once I find the time, I want to think about if these or its dependencies can be integrated into the existing libraries including antik mentioned by awesome-cl.

Personally, I've been relying on the stream-based method using py4cl/2, mostly because I did not - and perhaps do not - have the knowledge and time to dig into the CFFI based method. The limitation is that this would get you less than 10000 python interactions per second. That is sufficient if you will be running a long running python task - and I have successfully run trivial ML programs using it, but any intensive array processing gets in the way. For this later task, there are a few emerging libraries like numcl and array-operations without SIMD (yet), and numericals using SIMD. For reasons mentioned on the readme, I recently cooked up dense-arrays. This has interchangeable backends and can also use cl-cuda. But barring that, the developer overhead of actually setting up native-CFFI ecosystem is still too high, and I'm back to py4cl/2 for tasks beyond array processing.

I made this while running into code modularity issues with the numericals project I attempted last year; I did discover specialization-store, but found its goals in conflict with what I wanted to achieve; so I ended up investing in this.