dagger-for-github VS Dagger.jl

The Dockerized design also allows pipelines made with Dagger devkit to be run in every CI/CD runtime environment like, for example, Github Action (using the official Dagger Github Action from the marketplace). Furthermore, it can also be run independently of the architecture of the platform. The only requirement is the Docker ecosystem support. So it can be run on a managed runner (eg. Github Runners), a self-hosted runner, a local machine, a serverless compute instance, etc.

Fun dact, Crazy Max is the author of the Github Action for Dagger :) https://github.com/dagger/dagger-for-github

Firstly, I'll say that we already have work started to implement out-of-core directly in Dagger: https://github.com/JuliaParallel/Dagger.jl/pull/289.

With that PR in place, it should be possible to define a "storage device" which is backed by a database. I haven't had a chance to actually try this, since the PR still needs quite some work and testing, but it's definitely something on my radar!

I'm not sure that's necessarily the domain of a low-level package like CUDA.jl though (which I assume you're referring to). That kind of interface is more the domain of higher-level packages like https://github.com/JuliaParallel/Dagger.jl/ and to a lesser extent https://juliagpu.github.io/KernelAbstractions.jl/stable/. Moreover, the jury is still out on whether the built-in Distributed module is an ideal abstraction for every use-case (clusters, heterogeneous compute, etc.)

WRT Nx, my biggest question is how they'll crack the problem of still needing big balls of C++ and the shims everywhere to get acceleration. Creating a compiler that generates efficient GPU or other accelerator code is a massive research project with no clear winners, never mind the challenge of reconciling the very mutation-heavy needs of GPU compute with a mostly immutable language model.