pip VS pip-tools

Whenever you are working on a Python project that has external dependencies installed with pip, it is strongly recommended to first create a virtual environment.

Unfortunately that feature is easy to break: https://github.com/pypa/pip/issues/9644

I think I did my part https://github.com/pypa/pip/issues/6409

$ sudo pip install y$ sudo pip install yaml WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. ERROR: Could not find a version that satisfies the requirement yaml (from versions: none) ERROR: No matching distribution found for yaml

What are you implying will happen?

Using the build-in tools, you can save the exact versions of dependencies (i.e. a lock file) using "pip freeze >dependencies.txt". This should give you the exact same set of packages in two years' time.

If you want to be even more sure, you can also store hashes in the lock file. This has to be generated by a separate tools at the moment [1][2] but can be consumed by the built-in tools [3], so "pip install -r requirements.txt" is still all you need in two years' time.

[1] https://github.com/pypa/pip/issues/4732

[2] https://pip-tools.readthedocs.io/en/latest/#using-hashes

[3] https://pip.pypa.io/en/stable/topics/secure-installs/#hash-c...

Here’s the issue, https://github.com/pypa/pip/issues/11664. I think the idea would be to have some file/json description of environment that could be passed to pip to allow it to fully cross compile. They are open to supporting it just needs contributor to be found to implement it and go through review/discussion.

To be fair the only alternative is fixing Python, and even then you still would have to wait a good 5 years at least for all the old Python versions to dwindle.

It doesn't look like the fixing effort is progressing very quickly: https://github.com/pypa/pip/issues/8606

To their credit, at least they didn't close it "works as intended" which I imagine a lot of projects would.

Another good benchmark to trying to resolve apache-airflow[all]==1.10.13 using the state of PyPi on 2020-12-02, I give instructions here on how to reproduce that workflow: https://github.com/pypa/pip/issues/11836. Including a benchmark how how many extra packages your resolver should visit.

> Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts.

My friend, here is what you seek: https://github.com/jazzband/pip-tools

requirements.txt is flat because it's really the output of `pip freeze`. It's supposed to completely and exactly rebuild the environment. Unfortunately it's far too flexible and people abuse it by putting in only direct dependencies etc.

If you're writing packages, you don't need a requirements.txt at all, by the way. Package dependencies (only direct dependencies) live in pyproject.toml with the rest of the package config. requirements.txt (and pip tools) are only for when you want to freeze the whole environment, like for a server deployment.

For all my projects I found myself regenerating manual lock files using complex shell commands with pip-compile to get a reproducible environments across devices using a custom pre-install-command. I finally decided that instead of hacking together the same solution on all my projects I would build a plugin that handles this complexity for me.

Instead of venv, we are using pip-tools in this starter kit. pip-tools take things further in dependency management. Check out what pip-tools does in their official GitHub repo. In short, it helps your project find the best match for the dependent packages. For example, you might need two packages A and B in your project that requires same package C under the hood. But A requires any version of C from 1.0.1 to 1.0.10 and B requires any version of C from 1.0.7 to 1.0.15. Pip tools will automatically compile the version of 'C' that suits for both of your packages.

I've created a small project called just-pip-tools that combines pip-tools and just to manage Python dependencies in a layered approach. This isn't a magic bullet; it's a set of files you can adapt to your needs.

For small projects I recommend pip-tools. Just write packet list in requirements.in and pip-compile compile a requirements.txt with comments.

Take a look at pip-tools, great package. https://github.com/jazzband/pip-tools

pip-tools

I recommend generating a lockfile to document this information, as you might do with pip-tools.

Check out pip-tools [1] which does exactly that, albeit in a slightly more polished way.

[1]: https://github.com/jazzband/pip-tools