repairwheel VS cibuildwheel

I got into cross-compiling Python wheels (e.g., building macos wheels on linux and vice versa). Zig's `zig cc` does much of the heavy lifting, but one step in building a portable wheel is the "repair" process which vends native library dependencies into the wheel, necessitating binary patching (auditwheel does this for linux, delocate for macos).

I wanted to be able to do this cross platform, so I re-implemented ELF patching and Mach-O patching and adhoc signing in Python, and wrapped them into a tool called repairwheel: https://github.com/jvolkman/repairwheel

I'll plug some work I've been doing to (attempt to) enable cross compilation of Python wheels. I put together a small example [1] that builds the zstandard wheel, and can build macos wheels on linux and linux wheels on macos using zig cc.

macos wheels must still be adhoc signed (codesign) and binary patched (install_name_tool), so I re-implemented those functions in Python [2].

[1] https://github.com/jvolkman/bazel-pycross-zstandard-example

[2] https://github.com/jvolkman/repairwheel/tree/main/src/repair...

I mixed auditwheel, delocate, and delvewheel into a single tool called repairwheel and reimplemented all of the required external tools (patchelf, otool, codesign, etc.) in pure python.

It doesn't work with any version of the public API, Limited, Stable, or Unstable, because this is not a part of the API. It's more of an application of [Hyrum's Law](https://www.hyrumslaw.com/).

That said, assuming the structures themselves exist on the versions of Python you're targeting in a format compatible with whatever hacking you're doing on them, it's very easy to compile for lots of Python versions using [cibuildwheel](https://github.com/pypa/cibuildwheel) and the rest of the PyPA ecosystem.

I don't think the Limited API is very useful, as a practical matter for the common distribution methods you need the wheel to be built with the target Python version.

pypa/cibuildwheel: https://github.com/pypa/cibuildwheel :

> Example setup: To build manylinux, musllinux, macOS, and Windows wheels on GitHub Actions, you could use this .github/workflows/wheels.yml

> size of wheels you can upload is constrained by PyPi

I feel PyPi is pretty generous with their limits. You can even request more once you hit the ceiling, i think it’s around 60MB [1]. There are some wheels that are crazy large, tensorflow-gpu [2] are around 500MB each. I think there’s discussions out there to try and find ways of alleviating this problem on PyPi.

> difficult to support multiple versions across multiple operating systems, unless you provide a source distribution, which is then…

This can be a problem but I’ve found that recently the problem has improved quite a lot. You can create manylinux wheels for both x86, x64, and arm64 which cover pretty a lot of the Linux distributions using glibc. A musllinux tag was recently added to cover musl based distributions like Alpine. MacOS wheels support both x64, arm64, and can even be a universal2 wheel. Windows is still purely x86 or x64 for now but I’ve seen some people work on arm64 support support in CPython and once that’s in I’m sure PyPi won’t be too far around. There are also some great tools like cibuildwheel [3] that make building and testing these wheels pretty simple.

> Still a nightmare on Windows

I’m actually curious what is a nightmare about Windows. I found that Windows is probably the easiest of all the platforms to build and upload wheels for. You aren’t limited to a tiny subset of system libs, like you are on Linux, and building them is mostly the same process. Probably the hardest thing is ensuring you have tue correct vs build kit installed but that’s not insurmountable.

[1] https://pypi.org/help/#file-size-limit

[2] https://pypi.org/project/tensorflow-gpu/#files

[3] https://github.com/pypa/cibuildwheel

Click set up a workflow yourself to create a custom workflow. We can refer to the examples provided by cibuildwheel.