PDM VS pip

# See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint

PDM is a solution that allows for easy creation and management of python projects. Some of the key features that will improve the management of python projects include:

You'd need a proper dependencies management tool like PDM or Poetry to exhaustively resolve and lock down all the transitive dependencies if you want to have anything closed to reproducible build.

Are you working on a Python project that uses pdm for dependency management and dotenv for local environment variable and secrets management? Do you find it frustrating when CLI tools like pgcli don't automatically pick up your .env file, forcing you to resort to npm install -g dotenv-cli? I've got a more convenient solution for you!

I first learned about PDM from a blog post written by one of the PDM contributers. The post was about OOPifying argparse to allow for easy creation/modification of subcommands that exist as their own classes/files, and to avoid maintaining a single long script with an endless number of subparser.add_argument(...) lines.

That's more or less what PEP 582 plans to do, but it's been stalled and mired in discussions for years. The PDM tool went ahead and implemented it though if you want to use it: https://github.com/pdm-project/pdm

pdm – A modern Python package and dependency manager

pdm and PEP 582 enter the chat

I'm liking PDM for a while now. Quicker than Poetry and built according to the Python package spec in mind and not as an afterthought. While it was originally meant to work with PEP 582, it works with virtual environments too (now default).

https://github.com/pdm-project/pdm

Whenever you are working on a Python project that has external dependencies installed with pip, it is strongly recommended to first create a virtual environment.

Unfortunately that feature is easy to break: https://github.com/pypa/pip/issues/9644

I think I did my part https://github.com/pypa/pip/issues/6409

$ sudo pip install y$ sudo pip install yaml WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. ERROR: Could not find a version that satisfies the requirement yaml (from versions: none) ERROR: No matching distribution found for yaml

What are you implying will happen?

Using the build-in tools, you can save the exact versions of dependencies (i.e. a lock file) using "pip freeze >dependencies.txt". This should give you the exact same set of packages in two years' time.

If you want to be even more sure, you can also store hashes in the lock file. This has to be generated by a separate tools at the moment [1][2] but can be consumed by the built-in tools [3], so "pip install -r requirements.txt" is still all you need in two years' time.

[1] https://github.com/pypa/pip/issues/4732

[2] https://pip-tools.readthedocs.io/en/latest/#using-hashes

[3] https://pip.pypa.io/en/stable/topics/secure-installs/#hash-c...

Here’s the issue, https://github.com/pypa/pip/issues/11664. I think the idea would be to have some file/json description of environment that could be passed to pip to allow it to fully cross compile. They are open to supporting it just needs contributor to be found to implement it and go through review/discussion.

To be fair the only alternative is fixing Python, and even then you still would have to wait a good 5 years at least for all the old Python versions to dwindle.

It doesn't look like the fixing effort is progressing very quickly: https://github.com/pypa/pip/issues/8606

To their credit, at least they didn't close it "works as intended" which I imagine a lot of projects would.

Another good benchmark to trying to resolve apache-airflow[all]==1.10.13 using the state of PyPi on 2020-12-02, I give instructions here on how to reproduce that workflow: https://github.com/pypa/pip/issues/11836. Including a benchmark how how many extra packages your resolver should visit.