How to improve Python packaging, or why 14 tools are at least 12 too many

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • pip-audit

    Audits Python environments, requirements files and dependency trees for known security vulnerabilities, and can automatically fix them

  • sigstore-python

    A Sigstore client for Python

  • You could use `pip-compile` if you want full pinning. That's what we do on another project -- we use GitHub Actions with `pip-compile` to provide a fully frozen copy of the dependency tree for users who'd like that[1].

    In the context of `pip-audit`, that makes a little less sense: most of our dependencies are semantically versioned, and we'd rather users receive patches and fixes to our subdependencies automatically, rather than having to wait for us to release a corresponding fix version. Similarly, we expect users to install `pip-audit` into pre-existing virtual environments, meaning that excessive pinning will produce overly conservative dependency conflict errors.

    [1]: https://github.com/sigstore/sigstore-python/tree/main/instal...

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
  • Nuitka

    Nuitka is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, and 3.11. You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.

  • You probably already know about this, but Nuitka[0] is pretty great for building distributable Python apps.

    [0]: https://nuitka.net

  • pip

    The Python package installer

  • Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way".

    If you got to Python.org and follow through to the beginners guide [0] this is what's suggested:

    > There are several methods to install additional Python packages:

    > Packages can be installed via the standard Python distutils mode (python setup.py install).

    > Many packages can also be installed via the setuptools extension or pip wrapper, see https://pip.pypa.io/.

    That is so out of date and fundamentally confuses people coming to Python for the first time.

    The PyPA need to get buy in from Python Core to put one tool front and centre first. It needs to be like Rust with Cargo, literally the first thing you learn to use and core to all beginners guides.

    That's not to diminish the work of PyPA, you are all amazing I just want your work to be more obvious!

    0: https://docs.python.org/3/using/mac.html#installing-addition...

  • roadmap

    Public roadmap for the Poetry package manager (by python-poetry)

  • https://github.com/python-poetry/roadmap/issues/3

  • pragmatic-debian-packages

    Pragmatic Debian packaging

  • It depends what you mean by state-of-the-art, but it's possible with dh-virtualenv: https://github.com/vincentbernat/pragmatic-debian-packages/t... (not something that can become an official package as it goes against Debian policies)

  • publishing-python-packages

    Examples and exercises for Publishing Python Packages from Manning Books 🐍 📦 ⬆️

  • I don't agree with all the points from the article, but I do agree there is a depth of learning to be had about creating packages (and doing it repeatably/scalably). I wrote a book about creating Python packages that just came out: https://pypackages.com

    Even this book doesn't cover all options in each area, and it skips almost wholly over conda because I have no personal experience using it. conda and the work in the scientific community adds complexity both to the creation and the consumption side of packaging, and that's one area I'm not sure this post covers all the nuance of when considering how a "one size fits all" solution might work in practice.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • sampleproject

    A sample project that exists for PyPUG's "Tutorial on Packaging and Distributing Projects"

  • [2]: https://github.com/pypa/sampleproject/blob/main/pyproject.to...

  • pigar

    :coffee: A tool to generate requirements.txt for Python project, and more than that. (IT IS NOT A PACKAGE MANAGEMENT TOOL)

  • I am the author of pigar[1], and I am using Go a lot, Go has its problems too, but I am a fan of `import "url"` style import statement, developers can write code first, and sync the dependency later with `go mod tidy`.

    To fix problems in Python's world, Python's community should simplify the tools and cultivate a habit to declare the dependency first(maybe this should be mandatory) and use it later.

    [1]: https://github.com/damnever/pigar

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts