The future of Python build systems and Gentoo

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • janet

    A dynamic language and bytecode vm

  • build

    A simple, correct Python build frontend (by pypa)

  • Shouldn't https://github.com/pypa/build help? I don't think it has enough features yet but I was under the impression that was the distro solution.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
  • installer

    A low-level library for installing from a Python wheel distribution. (by pypa)

  • And probably https://github.com/pradyunsg/installer https://pypi.org/project/installer/ too, but that one isn't under the pypa org yet, given the author, it probably will be.

    All that being said, the deprecations and removals are being done way before the replacements are in place and tested and features/rough edges rounded out, which I guess is python tradition at this point.

    IMHO, the repos maintained by pypa should all be part of the core teams responsibility (e.g. pip is maintained by pypa, not core, but its installed as part of the python installer you download from the foundation's website...), along with solving building/installing/packaging/distributing as a whole. But the core team decided their volunteers didn't want to work on that (ok fair) and pypa stepped up to do it, but never really had enough manpower to tackle it well (ok also fair). And none of the tech companies ever thought "gee we rely so heavily on this awesome tool, maybe we should send them some money to keep the lights on..." So instead its just been a giant mess.

  • hpy

    HPy: a better API for Python

  • I've battled with Python's packaging systems for more than a decade. Every couple of years some new thing comes along promising to fix package management, but it always fails spectacularly on some important, mainstream package or use case. I suspect a lot of this complexity comes down to the significant degree in which the Python ecosystem depends on C: not only do we need to package Python programs but C programs as well, and some people think we should ship C source code and build on the target machine and others think we should ship pre-built C binaries and dynamically link against them on the target machine thus Python supports both mechanisms; however, both are perilous in the general case.

    And interestingly, Python depends so hard on C because CPython is very slow, and CPython is very slow because the C-extension API is virtually the whole of the CPython interpreter (many interesting optimizations to the interpreter would break compatibility in the ecosystem). So it certainly feels like all of Python's major problems come down to this decision to lean hard on C and specifically to make so much of the interpreter available for the C-extensions.

    The way out as far as I can tell is to define some narrower API surface (e.g., https://hpyproject.org), get the ecosystem to consolidate around that, deprecate the old API surface, and then make the requisite breaking optimizations such that the ecosystem can feasibly do more in native Python. This requires leadership; however, and this has not historically seemed to be Python's strong suite--the Python maintainers seem unable to drive out big, necessary changes like this (which is certainly not to say that leadership is easy, particularly when Python is so established in many respects).

    Personally, I've come to use Go for 99% of my Python use cases and it's been great. There are dramatically fewer C bindings, so the build/packaging tooling and performance are orders of magnitude better than in Python. Static typing works well out of the box, real static binaries are not only feasible but trivial (as opposed to Python where you try to build a zip file with your dependencies and the result is hundreds of megabytes and it's still missing the runtime, std libs, and various .so files). Further still, builds, tests, and every kind of tooling are far faster than with Python, and far simpler to install and manage. Unless you're doing data science, I don't think you'll regret the transition.

  • distros

    Calculate Linux Profiles (by gentoo-mirror)

  • This is on a tangent - TL;DR: I'm pretty convinced that the future of Gentoo is https://www.calculate-linux.org/

    After one too many times Python breaking changes got me to a point of Gentoo system that can't be updated (and it usually happens midway through bigger system/world update, so potentially many applications/tool also don't work) - I've switched to https://www.calculate-linux.org/

    It's still real Gentoo system, and unlike Sabayon Linux (many years ago) where you couldn't easily switch between emerge/portage and whatever Sabayon had - with Calculate it's all seamless. You can still do manual Stage3 tarball install, or just use Live image with GUI installer ...

    Most (like 99.95% percent) of packages come as pre-built binary, those remaining 0.05% is just packages where I had niche reasons to deviate from config (use flags) that Calculate used to build those binaries.

    And unlike `emerge -uND world` which can easily get stuck and even break (usually if you haven't updated system and toolchain in a while) - `cl-update` has been so good that several years passed and it didn't happen - well kind of.

    Combo of some python based apps being old/weird/whatever, and python breaking shit between versions - I do have multiple Python versions installed:

    * dev-lang/python

  • Cargo

    The Rust package manager

  • cargo itself does need TOML support to build because cargo itself is a Rust crate [0].

    But it's not a problem because you build a new version of cargo with an older version of cargo that of course can parse TOML.

    [0] https://github.com/rust-lang/cargo/blob/master/Cargo.toml

  • requests

    A simple, yet elegant, HTTP library.

  • No, this requests: https://github.com/psf/requests

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts