manylinux VS glibc_version_header

I use manylinux containers as the OS for compilation. It tries to ensure as much cross-os / libc / etc.. as much as possible for precompiled libraries. https://github.com/pypa/manylinux

Just to clarify for anyone who isn't aware, the "compiling issues", at least historically, have been that that Alpine uses musl, and PyPI's manylinux wheels are built against old glibc versions. So stuff like numpy that would trivially and quickly install from whl on glibc distros (like a bare-bones Ubuntu image) trigger compilations and the installation of build-only dependencies on Alpine.

That said, it looks like as of late-2021, at least some projects are offering musllinux wheels as well, per the discussion here: https://github.com/pypa/manylinux/issues/37 (not numpy, though: https://pypi.org/project/numpy/#files)

It's very hard. Incompatible glibc ABIs make this nigh impossible, there's a reason Steam installs a vcredistributable.dll for pretty much every game on Windows.

Look no further than the hoops you need jump through to distribute a Linux binary on PyPI [1]. Despite tons of engineering effort, and tons of hoop jumping from packagers, getting a non-trivial binary to run across all distros is still considered functionally impossible.

[1]: https://github.com/pypa/manylinux

As a user, if you build every python package from source, it's ok. But if you a maintainer of an OSS project and you need to publish binary packages for it, then you will hit the trouble. Binaries built on Ubuntu 20.04 can only support Ubuntu 20.04 and newer. So you'd better to choose an older Linux release to target broader users. Now most python packages choose CentOS 6 or 7. See https://github.com/pypa/manylinux/issues/1012 for more details. They need help!

I recently learned that Clang supports this kind of cross-compiling out of the box. https://mcilloni.ovh/2021/02/09/cxx-cross-clang/

The main difference is that Clang does not ship with headers/libraries for different platforms, as Zig appears to do. You need to give Clang a "sysroot" -- a path that has the headers/libraries for the platform you want to compile for.

If you create a bunch of sysroots for various architectures, you can do some pretty "easy" cross-compiling with just a single compiler binary. Docker can be a nice way of packaging up these sysroots (especially combined with Docker images like manylinux: https://github.com/pypa/manylinux). Gone are the days when you had to build a separate GCC cross-compiler for each platform you want to target.

Now you come and use manylinux to build. (https://github.com/pypa/manylinux) so you are based on the CentOS 7 toolchain (at best if you use manylinux2014) or Debian 9 toolchain (if you use manylinux_2_24).

I think the generally accepted way to do that would be a container image running a relatively old distribution. This is exactly what python packages do when they need to distribute binary packages on linux [0]. You are supposed to compile the package in a container (or VM) that runs CentOS 7 (or older if you want broader support), although now the baseline is moving gradually to Debian 9.

[0]: https://github.com/pypa/manylinux

One major headache with trying to run precompiled binaries on Linux is that if they were compiled using a newer version of glibc than the target machine, they won't be able to run. Back while working on Factorio, I was trying to get around this problem with endless Docker containers, but coworker Wheybags came up with a much solution to this, which is simply to, at compile time, link to the oldest compatible version of glibc: https://github.com/wheybags/glibc_version_header

If what you're doing works for you, great, but in case it stops working at some point (or if for some reason you need to build on a current-gen distro version), you could also consider using this:

https://github.com/wheybags/glibc_version_header

It's a set of autogenerated headers that use symbol aliasing to allow you to build against your current version of glibc, but link to the proper older versioned symbols such that it will run on whatever oldest version of glibc you select.

There are other approaches like https://github.com/wheybags/glibc_version_header or sysroots with older glibc, e.g. https://wiki.gentoo.org/wiki/Crossdev - you don't need your whole XP, just the the system libs to link against.

Sure, having a nice SDK where you can just specify the minimum vesion you want to support would be nice but who do you expect to develop such an SDK? GNU/glibc maintainers? They would rather you ship as source. Red Hat / SUSE / Canonical? They want you to target only their distro. Valve? They decided its easier to just provide an unchaning set of libraries since they need to support existing games that got things wrong anyway and already have a distribution platform to distribute such a base system along with the games without bundling it into every single one.

A few links gathered from a quick google search as a primer:

http://stevehanov.ca/blog/?id=97

https://www.evanjones.ca/portable-linux-binaries.html

https://insanecoding.blogspot.com/2012/07/creating-portable-...

https://rpg.hamsterrepublic.com/ohrrpgce/Portable_GNU-Linux_...

https://github.com/wheybags/glibc_version_header

In other words: there are a lot of steps and a lot of gotchyas to doing this that you're glossing over. Linux userland libraries are generally designed with the intention that an army of third-party maintainers will integrate all of this desperately developed software together and place it in a repo. Naturally every distribution wants to do things a little differently too, and they have a habit of changing it up every couple years. When you try to step out of that mold things unsurprisingly become more difficult. Whereas Windows, Mac, Android, etc. have been designed since the beginning not to require that sort of thing and it is consequently a much, much more straightforward process.

I'm curious why, since you seem to believe the process is so straight-forward, you think it is that so few people distribute a simple binary? Why were Flatpak and AppImage invented?

> Binaries compiled against today's glibc can fail to run on a machine that hasn't been updated since last week because they rely on a new / different symbol.

Note, however, that it is a Glibc bug (modulo Drepper’s temper) if the reverse happens: Glibc symbol versioning ensures that binaries depending on an old Glibc (only) will run on a new one. So the proper way to build a maximally-compatible Linux executable would be to build a cross toolchain targeting an old Glibc and compile your code with it. Unfortunately, the build system is hell and old Glibcs doesn’t compile without backported patches, so while I did try to follow in the footsteps of a couple of people[1–4], I did not succeed.

Mass-rebuilds still happen with other ecosystems, though. GHC-compiled Haskell libraries are fine-grained and not ABI-stable across compiler versions, so my Arch box regularly gets hit with a deluge of teensy library updates, and Arch is currently undergoing a massive Python rebuild (blocking all other Python package updates) behind the scenes as well.

[1]: https://github.com/wheybags/glibc_version_header (hack but easy and will probably work most of the time)