dh-virtualenv VS BinaryBuilder.jl

This is how Azure package functions, iirc. I think I used this https://github.com/spotify/dh-virtualenv for similar purpose some time ago.

I'm in a devops role where we actually reroll the Tensorflow whl in-house (to get a few tweaks like specific AVX flags turned on), but because the rest of our deployment is apt/debs, we then turn around and wrap that whl in a deb using Spotify's excellent dh-virtualenv:

https://github.com/spotify/dh-virtualenv

There's no expertise for Bazel in-house; when we run the build, it seems to fail all its cache hits and then spend 12-13h in total compiling, much of which appears to be recompiling a specific version of LLVM.

Every dependency is either vendored or pinned, including some critical things that have no ABI guarantees like Eigen, which is literally pinned to some a random commit, so that causes chaos when other binaries try to link up with the underlying Tensorflow shared objects:

https://github.com/tensorflow/tensorflow/blob/master/third_p...

And when you go down a layer into CUDA, there are even more support matrixes listing exact known sets of versions of things that work together:

https://docs.nvidia.com/deeplearning/tensorrt/support-matrix...

Anyway, I'm mostly just venting here. But the whole thing is an absurd nightmare. I have no idea how a normal distro would even begin to approach the task of unvendoring this stuff and shipping a set of normal packages for it all.

Rarely do I develop software at my host level, I'm almost always doing it inside of a container or a virtual machine (local or remote), and part of that is simple muscle memory to solve several problems that you've mentioned in your experience with Python, as well as other languages (bonus points for me that I was never plagued by a horrid npm bug). Those are problems that are easily solvable with Docker and a properly formatted requirements.txt (which should only be used in development — use pip/setup.py for proper deployment) and for the rare times when I do need to use Python at the host level of a Linux system, I use Spotify's dh-virtualenv.

Maybe I misunderstood the question, but this would be the point of CI/CD and containerization as far as industry standards are concerned. If you had to deploy at host level, then this would be the point of dh-virtualenv that’s tied in by your CI/CD system.

There are some efforts and the startup times are getting better with every release and there's BinaryBuilder.jl.

There is the Julia package https://github.com/JuliaPackaging/BinaryBuilder.jl which creates an environment that fakes being another, but with the correct compilers and SDKs . It's used to build all the binary dependencies

https://binarybuilder.org/. You can do it manually obviously, but this is easier.

> The main pain point is probably the lack of standard, multi-environment packaging solutions for natively compiled code.

Are you talking about something like BinaryBuilder.jl[1], which provides native binaries as julia-callable wrappers?

--

[1] https://binarybuilder.org

Julia did that for binary dependencies for a few years, with adapters for several linux platforms, homebrew, and for cross-compiled RPMs for Windows. It worked, to a degree -- less well on Windows -- but the combinatorial complexity led to many hiccups and significant maintenance effort. Each Julia package had to account for the peculiarities of each dependency across a range of dependency versions and packaging practices (linkage policies, bundling policies, naming variations, distro versions) -- and this is easier in Julia than in (C)Python because shared libraries are accessed via locally-JIT'd FFI, so there is no need to eg compile extensions for 4 different CPython ABIs (Julia also has syntactic macros which can be helpful here).

To provide a better experience for both package authors and users, as well as reducing the maintenance burden, the community has developed and migrated to a unified system called BinaryBuilder (https://binarybuilder.org) over the past 2-3 years. BinaryBuilder allows targeting all supported platforms with a single build script and also "audits" build products for common compatibility and linkage snafus (similar to some of the conda-build tooling and auditwheel). There was a nice talk at AlpineConf recently (https://alpinelinux.org/conf/) covering some of this history and detailing BinaryBuilder, although I'm not sure how to link into the video.

All that to say: it can work to an extent, but it has been tried various times before. The fact that conda and manylinux don't use system packages was not borne out of inexperience, either. The idea of "make binaries a distro packager's problem" sounds like a simplifying step, but that doesn't necessarily work out.