Distributions.jl VS clasp

If you look at Julia open source projects you'll see that the projects tend to have a lot more contributors than the Python counterparts, even over smaller time periods. A package for defining statistical distributions has had 202 contributors (https://github.com/JuliaStats/Distributions.jl), etc. Julia Base even has had over 1,300 contributors (https://github.com/JuliaLang/julia) which is quite a lot for a core language, and that's mostly because the majority of the core is in Julia itself.

This is one of the things that was noted quite a bit at this SIAM CSE conference, that Julia development tends to have a lot more code reuse than other ecosystems like Python. For example, the various machine learning libraries like Flux.jl and Lux.jl share a lot of layer intrinsics in NNlib.jl (https://github.com/FluxML/NNlib.jl), the same GPU libraries (https://github.com/JuliaGPU/CUDA.jl), the same automatic differentiation library (https://github.com/FluxML/Zygote.jl), and of course the same JIT compiler (Julia itself). These two libraries are far enough apart that people say "Flux is to PyTorch as Lux is to JAX/flax", but while in the Python world those share almost 0 code or implementation, in the Julia world they share >90% of the core internals but have different higher levels APIs.

If one hasn't participated in this space it's a bit hard to fathom how much code reuse goes on and how that is influenced by the design of multiple dispatch. This is one of the reasons there is so much cohesion in the community since it doesn't matter if one person is an ecologist and the other is a financial engineer, you may both be contributing to the same library like Distances.jl just adding a distance function which is then used in thousands of places. With the Python ecosystem you tend to have a lot more "megapackages", PyTorch, SciPy, etc. where the barrier to entry is generally a lot higher (and sometimes requires handling the build systems, fun times). But in the Julia ecosystem you have a lot of core development happening in "small" but central libraries, like Distances.jl or Distributions.jl, which are simple enough for an undergrad to get productive in a week but is then used everywhere (Distributions.jl for example is used in every statistics package, and definitions of prior distributions for Turing.jl's probabilistic programming language, etc.).

...so the blog post you've posted 4 times contains a list of issues the author filed in 2020-2021... and at least for the handful I clicked, they indeed have (long) been sorted. e.g., Filed Dec 18th 2020, closed Dec 20th

Well, out of the issues mentioned, the ones still open can be categorized as (1) aliasing problems with mutable vectors https://github.com/JuliaLang/julia/issues/39385 https://github.com/JuliaLang/julia/issues/39460 (2) not handling OffsetArrays correctly https://github.com/JuliaStats/StatsBase.jl/issues/646, https://github.com/JuliaStats/StatsBase.jl/issues/638, https://github.com/JuliaStats/Distributions.jl/issues/1265 https://github.com/JuliaStats/StatsBase.jl/issues/643 (3) bad interaction of buffering and I/O redirection https://github.com/JuliaLang/julia/issues/36069 (4) a type dispatch bug https://github.com/JuliaLang/julia/issues/41096

So if you avoid mutable vectors and OffsetArrays you should generally be fine.

As far as the argument "Julia is really buggy so it's unusable", I think this can be made for any language - e.g. rand is not random enough, Java's binary search algorithm had an overflow, etc. The fixed issues have tests added so they won't happen again. Maybe copying the test suites from libraries in other languages would have caught these issues earlier, but a new system will have more bugs than a mature system so some amount of bugginess is unavoidable.

See the Distributions.jl package for an example .jl file structure: https://github.com/JuliaStats/Distributions.jl/tree/master/src

Structure your program around your domain specific constrains, e.g if you look at Distributions.jl they have folders for univariate/multivariate or discrete/continuous with a file per distribution containing the struct + all its methods :

I accidentally a Common Lisp that interoperates with C++ (https://github.com/clasp-developers/clasp.git). We would also like to move beyond BDWGC and Whiffle looks interesting. I will reach out to you and maybe we can chat about it.

Clasp might be such a language, it seems.

https://github.com/clasp-developers/clasp

/u/jeaye are you aware of CLASP? https://github.com/clasp-developers/clasp https://www.youtube.com/watch?v=mbdXeRBbgDM

Sounds to me like CLASP; it automatically exports C++ objects to be used from Common Lisp also via llvm.

Now, the difference of compiling speed of SBCL and CCL is not so big. Look at cl-benchmark, LispWorks is really fast, CCL is on par with Allegro, SBCL is close to CCL. Or https://github.com/clasp-developers/clasp/wiki/Relative-Compile-Performance-of-clasp, it depends on specific project (SBCL sometimes faster, slower, alike), overall difference is not big.

So..

"Why do you want to make Lisp more popular? If you were sucessful, what would be different in the world, and why is that desirable to you?"

Normally at this point I'd listen to the response, and ask more questions based on that. That would wind up with a very, very deep thread, so I'll break a cardinal rule and pre-guess at some answers.

This kind of question comes up pretty frequently. In many cases, I suspect the motivation behind the question is "Wow! Here's this cool tool I've discovered. I want to make something really useful with it. I want to do it as part of a community effort; share my excitement with others, share in their excitement, and know that what I'm making is useful because others find it desirable and are excited by it." The field could be cooking, sports, old machine tools, tiny homes, or demo scene. Its the fundemental driver for most content on HN, YouTube, Instructables, and such. It is a Good Thing.

If that is your motivator, then my suggestion is to find something that bugs you and fix it. You've already decided you're only interested in code, not other aspects. You said you preferred vim, but the emacs ecosystem has a very rich set of sharp edges that need filing off, and a rich set of tools with which to attack them.

One example: even after 50 years there's no open IDE which allows you to easily globally rename a Lisp identifier. I don't know about LispWorks or other proprietary environments, but you can't in emacs or vim do a right-click on "foo" in "(defun foo ()...)" and select a command which automatically renames it in all invocations. [Queue lots of "but you can..." replies here.] I don't think vim is up to the task of doing this internally. It would be possible in emacs; but would require a huge effort with lots of help from other people. If you emerged alive from that rabbit warren you'd join the company of Certified "How Hard Could it Be?" Mad Scientists such as Dr. "I just want to draw molecules" Meister [1] and "Wouldn't an OS in Lisp be Cool" Froggey [2].

[1] https://github.com/clasp-developers/clasp

[2] Mezzano https://github.com/froggey/Mezzano

But also, there's a reason why most implementations readily make an effort to provide interoperability tools with a variety of runtimes. Clasp much like ABCL gives access to a whole library of other libraries trivially wrapped to interoperate with at little to no performance to cost (depending on how thin you make the wrappers, mainly).