Bundling binary tools in Python wheels

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • cibuildwheel

    🎡 Build Python wheels for all the platforms with minimal configuration.

    > size of wheels you can upload is constrained by PyPi

    I feel PyPi is pretty generous with their limits. You can even request more once you hit the ceiling, i think it’s around 60MB [1]. There are some wheels that are crazy large, tensorflow-gpu [2] are around 500MB each. I think there’s discussions out there to try and find ways of alleviating this problem on PyPi.

    > difficult to support multiple versions across multiple operating systems, unless you provide a source distribution, which is then…

    This can be a problem but I’ve found that recently the problem has improved quite a lot. You can create manylinux wheels for both x86, x64, and arm64 which cover pretty a lot of the Linux distributions using glibc. A musllinux tag was recently added to cover musl based distributions like Alpine. MacOS wheels support both x64, arm64, and can even be a universal2 wheel. Windows is still purely x86 or x64 for now but I’ve seen some people work on arm64 support support in CPython and once that’s in I’m sure PyPi won’t be too far around. There are also some great tools like cibuildwheel [3] that make building and testing these wheels pretty simple.

    > Still a nightmare on Windows

    I’m actually curious what is a nightmare about Windows. I found that Windows is probably the easiest of all the platforms to build and upload wheels for. You aren’t limited to a tiny subset of system libs, like you are on Linux, and building them is mostly the same process. Probably the hardest thing is ensuring you have tue correct vs build kit installed but that’s not insurmountable.

    [1] https://pypi.org/help/#file-size-limit

    [2] https://pypi.org/project/tensorflow-gpu/#files

    [3] https://github.com/pypa/cibuildwheel

  • postgresql-wheel

    A Python wheel containing PostgreSQL

    This is also the basis for the postgresql-wheel package, a Python wheel that contains an entire local PostgreSQL binary installation:

    https://github.com/michelp/postgresql-wheel

    Out of sheer laziness I used CFFI to encapsulate the build, even though there is no dynamic linking involved. There's probably a simpler way but this approach has worked very well for disposable database tests.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • sigstore-python

    A codesigning tool for Python packages

    You're right, both the infrastructure and metadata for cryptographic signatures on Python packages (both wheels and sdists) isn't quite there yet.

    At the moment, we're working towards the "e2e" scheme you've described by adding support for Sigstore[1] certificates and signatures, which will allow any number of identities (including email addresses and individual GitHub release workflows) to sign for packages. The integrity/availability of those signing artifacts will in turn be enforced through TUF, like you mentioned.

    You can follow some of the related Sigstore-in-Python work here[2], and the ongoing Warehouse (PyPI) TUF work here[3]. We're also working on adding OpenID Connect token consumption[4] to Warehouse itself, meaning that you'll be able to bootstrap from a trusted GitHub workflow to a PyPI release token without needing to share any secrets.

    [1]: https://www.sigstore.dev/

    [2]: https://github.com/sigstore/sigstore-python

    [3]: https://github.com/pypa/warehouse/pull/10870

    [4]: https://github.com/pypa/warehouse/pull/11272

  • warehouse

    The Python Package Index

    You're right, both the infrastructure and metadata for cryptographic signatures on Python packages (both wheels and sdists) isn't quite there yet.

    At the moment, we're working towards the "e2e" scheme you've described by adding support for Sigstore[1] certificates and signatures, which will allow any number of identities (including email addresses and individual GitHub release workflows) to sign for packages. The integrity/availability of those signing artifacts will in turn be enforced through TUF, like you mentioned.

    You can follow some of the related Sigstore-in-Python work here[2], and the ongoing Warehouse (PyPI) TUF work here[3]. We're also working on adding OpenID Connect token consumption[4] to Warehouse itself, meaning that you'll be able to bootstrap from a trusted GitHub workflow to a PyPI release token without needing to share any secrets.

    [1]: https://www.sigstore.dev/

    [2]: https://github.com/sigstore/sigstore-python

    [3]: https://github.com/pypa/warehouse/pull/10870

    [4]: https://github.com/pypa/warehouse/pull/11272

  • auditwheel

    Auditing and relabeling cross-distribution Linux wheels.

    In a similar vein: Python projects can use `auditwheel` to automatically relocate (fixup RPATHs) and vendor their "system" dependencies, such as a specific version of `zlib` or `libffi`[1].

    [1]: https://github.com/pypa/auditwheel

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts