hydra VS adcomp

See https://github.com/NixOS/hydra/issues/838 for making content-addressed derivations supported by hydra.nixos.org. At that point, we can actually try out the XP feature at scale.

Also see https://github.com/NixOS/nix/issues/8919 for this accepted RFC

Once those things are done, we can get back to merging in the IPFS code.

Now that there is an Nix team and I am on it, there is much, much less of an issue of these experiments being caught in limbo :).

https://github.com/nix-community/infra, Community project builds https://github.com/NixOS/hydra, NixOS build server

Nix is pretty cool, and I would say comparisons to Earthly are apt. I may tackle that in a follow-up. If you did a monorepo setup written in nix and then used something like Hydra for building, it might be a pretty nice solution.

Nix seems great for build servers. This is a great introduction to the motivations behind it.

I'm not sold on using it for managing developer environments (another use case it is often used for). It "solves" the problem that developers might be using different versions of libraries or compilers on their machines... but it comes at the cost of having to learn a whole new programming language, a configuration language, a whole new jargon, and workflow. It's a bit like using Docker as a development environment. It introduces a non-trivial amount of friction.

Some folks get excited about package management and configuration. Personally I don't care for it enough to over-come such a high learning curve. And I don't particularly like the workflow it enforces.

However it is pretty great for reproducible CI/CD systems like Hydra: https://github.com/NixOS/hydra

Somewhat related to this, is there a good way to install something from a flake inside the configuration.nix? For example, the hydra flake, since it includes many derivations for dependencies that are not part of nixpkgs (or are at the wrong versions).

Are any of your R users TMB users? Because I ran into that in multiple setups (Guix SD and Guix on foreign distros) where only a specific ordering worked even when using guix shell --pure. This was back in the R 4.1.2 days. In theory it should be impossible and yet I ran into it. No I was not mixing packages installed via. install.packages.

Right now with the latest version of Guix and R 4.2.1 TMB is not usable. Try running:

"guix shell --container r r-tmb make gcc-toolchain gfortran-toolchain"

then try running the linreg.R (with the corresponding cpp file, or any of the examples) example from https://github.com/kaskr/adcomp/tree/master/tmb_examples

There is no free lunch:).

I remember spending a summer using Template Model Builder (TMB), which is a useful R/C++ automatic differentiation (AD) framework, for working with accelerated failure time models. For these models, the survival to time T given covariates X is defined by S(t|X) = P(T>t|X) = S_0(t exp(-beta^T X)) for baseline survival S_0(t). I wanted to use splines for the baseline survival and then use AD for gradients and random effects. Unfortunately, after implementing the splines in template C++, I found a web page entitled "Things you should NOT do in TMB" (https://github.com/kaskr/adcomp/wiki/Things-you-should-NOT-d...) - which included using if statements that are based on coefficients. In this case, the splines for S_0 depend on beta, which is this specific excluded case:(. An older framework (ADMB) did not have this constraint, but dissemination of code was more difficult. Finally, PyTorch did not have an implementation of B-splines or an implementation for Laplace's approximation. Returning to my opening comment, there is no free lunch.