Pkg.jl VS rr

There was a "bug" (or just unhandled caching case) that effected the Pluto notebook system that required precompilation each time. This is because Pluto notebooks kept a manifest (so they always instantiated with the same packages every time for full reproducibility) and the instantiation of that manifest triggered not just package running but also precompilation. That was fixed in https://github.com/JuliaLang/Pkg.jl/pull/3378, with a larger discussion in https://discourse.julialang.org/t/first-pluto-notebook-launc.... That should largely remove this issue as in included in the v1.9 release (it was first in v1.9-RC2 IIRC).

The closest thing I got to is this and I don't even understand what they are saying.

Julia's compiler is made to be extendable. GPUCompiler.jl which adds the .ptx compilation output for example is a package (https://github.com/JuliaGPU/GPUCompiler.jl). The package manager of Julia itself... is an external package (https://github.com/JuliaLang/Pkg.jl). The built in SuiteSparse usage? That's a package too (https://github.com/JuliaLang/SuiteSparse.jl). It's fairly arbitrary what is "external" and "internal" in a language that allows that kind of extendability. Literally the only thing that makes these packages a standard library is that they are built into and shipped with the standard system image. Do you want to make your own distribution of Julia that changes what the "internal" packages are? Here's a tutorial that shows how to add plotting to the system image (https://julialang.github.io/PackageCompiler.jl/dev/examples/...). You could setup a binary server for that and now the first time to plot is 0.4 seconds.

Julia's arrays system is built so that most arrays that are used are not the simple Base.Array. Instead Julia has an AbstractArray interface definition (https://docs.julialang.org/en/v1/manual/interfaces/#man-inte...) which the Base.Array conforms to, and many effectively standard library packages like StaticArrays.jl, OffsetArrays.jl, etc. conform to, and thus they can be used in any other Julia package, like the differential equation solvers, solving nonlinear systems, optimization libraries, etc. There is a higher chance that packages depend on these packages then that they do not. They are only not part of the Julia distribution because the core idea is to move everything possible out to packages. There's not only a plan to make SuiteSparse and sparse matrix support be a package in 2.0, but also ideas about making the rest of linear algebra and arrays themselves into packages where Julia just defines memory buffer intrinsic (with likely the Arrays.jl package still shipped with the default image). At that point, are arrays not built into the language? I can understand using such a narrow definition for systems like Fortran or C where the standard library is essentially a fixed concept, but that just does not make sense with Julia. It's inherently fuzzy.

This is exacerbated by the fact that Julia's Pkg.jl does not yet support conditional/optional dependencies [0]. A lot of these meta packages tend to pull everything but the kitchen sink.

[0]: https://github.com/JuliaLang/Pkg.jl/issues/1285

The LTS release is over two years old, and Julia has received a lot of developer attention since then, resulting in new features and performance improvements that tutorial authors don't want to do without. You can safely use the latest stable release (v1.5.3), although you may also want to apply the Git registry fix (https://github.com/JuliaLang/Pkg.jl/issues/2014#issuecomment-730676631) for further improvements in download/setup speed.

I think this tool must share a lot techniques and use cases with rr. I wonder how it compares in various aspects.

https://rr-project.org/

rr "sells" as a "reversible debugger", but it obviously needs the determinism for its record and replay to work, and AFAIK it employs similar techniques regarding system call interception and serializing on a single CPU. The reversible debugger aspect is built on periodic snapshotting on top of it and replaying from those snapshots, AFAIK. They package it in a gdb compatible interface.

Hermit also lists record/replay as a motivation, although it doesn't list reversible debugging in general.

Interesting. Perhaps you can inspect the disassembly of the function in question when using Graal and HotSpot. It is likely related to that.

Another debugging technique we use for heisenbugs is to see if `rr` [1] can reproduce it. If it can then that's great as it allows you to go back in time to debug what may have caused the bug. But `rr` is often not great for concurrency bugs since it emulates a single-core machine. Though debugging a VM is generally a nightmare. What we desperately need is a debugger that can debug both the VM and the language running on top of it. Usually it's one or the other.

> In general I’d argue you haven’t fixed a bug unless you understand why it happened and why your fix worked, which makes this frustrating, since every indication is that the bug exists within proprietary code that is out of my reach.

Were you using Oracle GraalVM? GraalVM community edition is open source, so maybe it's worth checking if it is reproducible in that.

[1]: https://github.com/rr-debugger/rr

https://rr-project.org/ had the same problem. They use the retired conditional branch counter instead of instruction counter, and then instruction steeping until at the correct address.

That'll work great for your Distributed QSort Incorporated startup, where the only product is a sorting algorithm.

Formal software verification is very useful. But what can be usefully formalized is rather limited, and what can be formalized correctly in practice is even more limited. That means you need to restrict your scope to something sane and useful. As a result, in the real world running thousands of tests is practically useful. (Well, it depends on what those tests are; it's easy to write 1000s of tests that either test the same thing, or only test the things that will pass and not the things that would fail.) They are especially useful if running in a mode where the unexpected happens often, as it sounds like this system can do. (It's reminiscent of rr's chaos mode -- https://rr-project.org/ linking to https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo... )

The approach of check-pointing computation such that it is resumable and restartable sounds similar to a time-traveling debugger, like rr or WinDbg:

https://rr-project.org/

https://learn.microsoft.com/windows-hardware/drivers/debugge...

https://github.com/rr-debugger/rr?tab=readme-ov-file#system-...