Python Best Practices for a New Project in 2021

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • pypiserver

    Minimal PyPI server for uploading & downloading packages with pip/easy_install

    tarballs, rpms or debs also work.

    One should probably run their own package server like https://github.com/pypiserver/pypiserver

    All of that said, containers are nice because you have a log of what is running, easy to transport and coordinate.

    When you use Go and Rust over Python, does the use of Docker disappear? What replaces it?

  • conda

    A system-level, binary package and environment manager running on all major operating systems and platforms.

    Vonda has been a lifesaver for me in the past, but it got so slow in ~2019 (minutes+ to resolve dependencies) that I've switched back to pip whenever possible. Maybe things have been resolved now though?

    E.g. https://github.com/conda/conda/issues/8087, https://www.anaconda.com/blog/understanding-and-improving-co...

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • Poetry

    Python packaging and dependency management made easy

    Indeed, why Poetry? Poetry At first glance looks and feels good (and I even used it for a while) until you hit a weird bug and find that’s it’s one of the 989 open issues https://github.com/python-poetry/poetry/issues

    So it’s back to plain old venv + requirements.txt for me

  • template-python-hello-world

    :triangular_ruler: Python Hello World | Minimal template for Python development

    The current work project[1] has all of these: Pyenv, Poetry, Pytest, pytest-cov with 100% branch coverage, pre-commit, Pylint rather than Flake8, Black, mypy (with a stricter configuration than recommended here), and finally isort. These are all super helpful.

    There's also a simpler template repo[2] with almost all of these.

    [1] https://github.com/linz/geostore/

    [2] https://github.com/linz/template-python-hello-world

  • miniforge

    A conda-forge distribution.

    Does it not work with miniforge/mambaforge? (https://github.com/conda-forge/miniforge)

    I use this with my m1 mac, and it works great.

  • pyright

    Static Type Checker for Python

  • devpi

    Python PyPi staging server and packaging, testing, release tool

    > One should probably run their own package server like https://github.com/pypiserver/pypiserver

    Never used pypiserver but I’ve had a good experience with https://github.com/devpi/devpi

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • flake8-alphabetize

    Python style checker for import statements, __all__ lists and exception lists, implemented as a flake8 plugin

    I think you mean Flake8 rather than Sense8? Anyway, gives me a chance to plug Flake8 Alphabetize, a Flake8 plugin for import ordering https://github.com/tlocke/flake8-alphabetize

  • mamba

    The Fast Cross-Platform Package Manager (by mamba-org)

    > Conda has been a lifesaver for me in the past, but it got so slow in ~2019

    This is why mamba [0] was created. It is a C++ reimplementation of conda for much better performance. mamba is a drop-in replacement of conda and can operate on the same anaconda, condaforge (and mambaforge) repositories.

    [0] https://github.com/mamba-org/mamba

  • pip

    The Python package installer

    pip has 886 open issues[0], I am not sure the issue total is much of a measuring stick in this case.

    my problem is that venv + requirements.txt is just plain simpler to use.

    [0] https://github.com/pypa/pip/issues

  • pip-tools

    A set of tools to keep your pinned Python dependencies fresh.

    With pip-tools.

    https://github.com/jazzband/pip-tools

    And you can still use standard setup.cuff and pip install -e unlike Poetry. Also, much faster.

  • Zappa

    Serverless Python (by zappa)

    Heroku, yes I would push for that. Its expensive, but it'll save you an entire devop function right up til you hit the 30 people mark.

    Elastic beanstalk is just a horrid dev environment. Lots of waiting, lots of non-obvious options, and very little reward. I would personally push for lambda and zappa (https://github.com/zappa/Zappa) for python, as it seems to be much easier to deploy and debug.

  • flynt

    A tool to automatically convert old string literal formatting to f-strings

    That is a great write-up! One extra bit I'd recommend to this list is using https://github.com/ikamensh/flynt to convert string format into f-strings. It requires Python 3.6.

  • pyenv

    Simple Python version management

    Because pyenv [1] and venv [2] are entirely different things that do two completely different jobs?

    [1]: https://github.com/pyenv/pyenv

  • pydantic

    Data validation using Python type hints

    Man, developing python with pydantic and "mypy --strict" (I follow pydantic's config [0] where I can) is such Type 2 Fun. It feels like a totally different language. Yeah it takes a little more time at first but then type inference and autocomplete starts to kick in and then you're screaming fast. And you "compile" it and everything just works. No hunting down edge cases or tracebacks cause you forgot to catch a None. I find it super satisfying. Much easier to stay in flow state when you aren't having to stop every few minutes to test stuff and dig through tracebacks.

    [0] https://github.com/samuelcolvin/pydantic/blob/master/setup.c...

  • helm

    The Kubernetes Package Manager

    > running a signed binary doesn't save you.

    True, it does not. I don't recommend downloading (random) binary installers and running them either.

    With eg Linux isos, you typically already trust the signing key for your os updates.

    But unless you are vigilant about your ssl root certs, you'll easily allow a lot of malicious and incompetent services to potentially intercept most of your ssl traffic... (due to there being many trusted roots by default).

    > if someone has overtaken a host and replaced the binaries

    This again depend on who and how the binaries are signed, and how the signatures are trusted. Typical windows (and Mac?) setups will gobble up any signature. But if you do check who signs the binaries - then the signing key will easily be the most secure part of the system - a compromised ftp/web site allow hosting malicious binaries, but typically not grant access to the signing key.

    With letsencrypt a hacked web site will typically have access to a valid ssl cert - no need to further compromise mx/mail records or gain access to a business phone number etc.

    A ascii-armor signed shell script can be distributed safely via a paste-bin. Unfortunately there's no good automatic/standard way to do so. Or rather no standard tool to prompt to trust the signing key - and then run the script - beyond basic gpg --search-key --key-server.. + gpgv.

    Maybe signed git repos would be easiest - but I don't know how easy it is to limit which keys are trusted - if it's possible at all?

    The helm project does a little dance to try and verify downloads - but for all the effort it pretty much amounts to trusting the script, not the keys/signatures:

    https://github.com/helm/helm/blob/v3.6.2/scripts/get-helm-3#...

    I was hopeful sequoia might help - but apparently its sqv tool is even worse than gpgv - neither can handle an ascii armored public key, and sqv can only handle detached signatures.

    And just for completeness - a reminder that any cut'n'paste in the terminal is a bad idea:

  • PyInstaller

    Freeze (package) Python programs into stand-alone executables

    If this is a company project, a REST API or task service of some kind, then you are probably using Docker in the year 2021.

    If this is anything else, or if you work for a shop that hasn't embraced containerization, then you use PyInstaller (http://www.pyinstaller.org) to bundle your application. Either into a directory that contains your full Python virtual environment (only 5-10 megs!), or into a single executable file.

    The latter is most convenient for a Go/Rust type experience. But the former will startup faster, because that single-file executable has to first uncompress itself to the system temp directory.

  • geospatial-data-lake

    Discontinued Central storage, management and access for important geospatial datasets

    The current work project[1] has all of these: Pyenv, Poetry, Pytest, pytest-cov with 100% branch coverage, pre-commit, Pylint rather than Flake8, Black, mypy (with a stricter configuration than recommended here), and finally isort. These are all super helpful.

    There's also a simpler template repo[2] with almost all of these.

    [1] https://github.com/linz/geostore/

    [2] https://github.com/linz/template-python-hello-world

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts