hydra VS mach-nix

See https://github.com/NixOS/hydra/issues/838 for making content-addressed derivations supported by hydra.nixos.org. At that point, we can actually try out the XP feature at scale.

Also see https://github.com/NixOS/nix/issues/8919 for this accepted RFC

Once those things are done, we can get back to merging in the IPFS code.

Now that there is an Nix team and I am on it, there is much, much less of an issue of these experiments being caught in limbo :).

https://github.com/nix-community/infra, Community project builds https://github.com/NixOS/hydra, NixOS build server

Nix is pretty cool, and I would say comparisons to Earthly are apt. I may tackle that in a follow-up. If you did a monorepo setup written in nix and then used something like Hydra for building, it might be a pretty nice solution.

Nix seems great for build servers. This is a great introduction to the motivations behind it.

I'm not sold on using it for managing developer environments (another use case it is often used for). It "solves" the problem that developers might be using different versions of libraries or compilers on their machines... but it comes at the cost of having to learn a whole new programming language, a configuration language, a whole new jargon, and workflow. It's a bit like using Docker as a development environment. It introduces a non-trivial amount of friction.

Some folks get excited about package management and configuration. Personally I don't care for it enough to over-come such a high learning curve. And I don't particularly like the workflow it enforces.

However it is pretty great for reproducible CI/CD systems like Hydra: https://github.com/NixOS/hydra

Somewhat related to this, is there a good way to install something from a flake inside the configuration.nix? For example, the hydra flake, since it includes many derivations for dependencies that are not part of nixpkgs (or are at the wrong versions).

Another way if the above doesn't work is to use mach-nix:https://github.com/DavHau/mach-nix

For python, I can highly recommend mach-nix. Makes it really easy to also keep a requirements.txt to stay compatible with non-nix-users.

The ESP32 toolchain is quite cumbersome to install - even under NixOS. The wiki tells you to create a shell.nix that creates a python virtual env at the first execution. I used DavHau's mach-nix to create esp-idf.nix and subsequently micropython-esp32.nix.

mach-nix and friends are pretty good, but I've found that the expectations of all the commonly-used data science stuff is pretty antithetical to the Nix Way. I'm not sure if it's still the case, but last I tried, Hydra wasn't building CUDA (since it's non-free), so I had to compile that and e.g. TensorFlow and PyTorch. Very painful, even on a beefy system.

I don't do much Python, but usually everything works just fine with mach-nix. I would try something like this: ``` { pkgs ? import (fetchTarball "https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz") { } }:

There are some sifferent/new tools for adding your own Python packages these days. It's still not truly solved, but one of these new package generation tools might serve your better:

poetry2nix: https://github.com/nix-community/poetry2nix

dream2nix: https://nix-community.github.io/dream2nix/guides/getting-sta...

mach-nix: https://github.com/DavHau/mach-nix

pip2nix: https://github.com/nix-community/pip2nix

pynixify: https://github.com/cript0nauta/pynixify

The tools available to you at the time (pypi2nix and maybe python2nix, if it was a long time ago) have been abandoned in favor of the newer tools, I think chiefly poetry2nix but I'm not sure.

There's still the Nixpkgs buildPythonPackage stuff, I think, if your goal is to upstream a lib into Nixpkgs. But if you just want to build your own Python applications and vendorize the deps (e.g., for work), you might try one of the tools above, which weren't available 3+ years ago. Maybe Nixy Python users and developers can reply with some of their experiences using those tools :)

Have a look at mach-nix which is a small utility library for nix to build Python packages declaratively.

final: prev: { xonsh = prev.xonsh.overrideAttrs (old: rec { version = "0.12.4"; src = final.fetchFromGitHub { owner = "xonsh"; repo = "xonsh"; rev = version; sha256 = "0kdps0gf0767zy0fs6qn39rv4z3x7ck0qz1pzx6962593171yk8b"; }; propagatedBuildInputs = prev.xonsh.propagatedBuildInputs ++ [final.python3Packages.virtualenv]; }); python39 = prev.python39.override { self = prev.python39; packageOverrides = python_final: python_prev: { prompt-toolkit = python_prev.prompt-toolkit.overrideAttrs (old: rec { version = "3.0.29"; src = final.python3Packages.fetchPypi { pname = "prompt_toolkit"; inherit version; sha256 = "sha256-vWQPYOjOzXTw3CSXE9QzrOLdxitl7gf5bTWOCxUrbqc="; }; }); }; }; # Using mach-nix to fetch unpackaged xontrib plugins # adapted from https://github.com/NixOS/nixpkgs/issues/75786#issuecomment-873654103 mach-nix = import (builtins.fetchGit { url = "https://github.com/DavHau/mach-nix/"; ref = "refs/tags/3.4.0"; }) { pkgs = final; }; xonsh_pyenv = final.mach-nix.mkPython { requirements = '' xontrib-fzf-widgets xonsh-direnv ''; }; xonsh_with_plugins = final.xonsh.overrideAttrs (old: { propagatedBuildInputs = old.propagatedBuildInputs ++ final.xonsh_pyenv.python.pkgs.selectPkgs final.xonsh_pyenv.python.pkgs; }); }