packaging VS dh-virtualenv

> You can't use wheels

musl wheel support was added some time ago:

https://github.com/pypa/packaging/releases/tag/21.0

No idea how many packages actually build musl wheels though. From a quick glance at least cryptography and lxml has musl wheels.

Python supports Alpine just fine; indeed, the standard Docker Python images include Alpine variants.

If you mean binary wheels, PyPA packaging added support for Alpine wheels a few months ago (https://github.com/pypa/packaging/pull/411), and auditwheel support for the same just shipped today.

You can use the packaging module to compare versions. It's used by setuptools so you probably have it already installed. The the rest is pretty straightforward imho:

This is how Azure package functions, iirc. I think I used this https://github.com/spotify/dh-virtualenv for similar purpose some time ago.

I'm in a devops role where we actually reroll the Tensorflow whl in-house (to get a few tweaks like specific AVX flags turned on), but because the rest of our deployment is apt/debs, we then turn around and wrap that whl in a deb using Spotify's excellent dh-virtualenv:

https://github.com/spotify/dh-virtualenv

There's no expertise for Bazel in-house; when we run the build, it seems to fail all its cache hits and then spend 12-13h in total compiling, much of which appears to be recompiling a specific version of LLVM.

Every dependency is either vendored or pinned, including some critical things that have no ABI guarantees like Eigen, which is literally pinned to some a random commit, so that causes chaos when other binaries try to link up with the underlying Tensorflow shared objects:

https://github.com/tensorflow/tensorflow/blob/master/third_p...

And when you go down a layer into CUDA, there are even more support matrixes listing exact known sets of versions of things that work together:

https://docs.nvidia.com/deeplearning/tensorrt/support-matrix...

Anyway, I'm mostly just venting here. But the whole thing is an absurd nightmare. I have no idea how a normal distro would even begin to approach the task of unvendoring this stuff and shipping a set of normal packages for it all.

Rarely do I develop software at my host level, I'm almost always doing it inside of a container or a virtual machine (local or remote), and part of that is simple muscle memory to solve several problems that you've mentioned in your experience with Python, as well as other languages (bonus points for me that I was never plagued by a horrid npm bug). Those are problems that are easily solvable with Docker and a properly formatted requirements.txt (which should only be used in development — use pip/setup.py for proper deployment) and for the rare times when I do need to use Python at the host level of a Linux system, I use Spotify's dh-virtualenv.

Maybe I misunderstood the question, but this would be the point of CI/CD and containerization as far as industry standards are concerned. If you had to deploy at host level, then this would be the point of dh-virtualenv that’s tied in by your CI/CD system.