pip VS asdf

Whenever you are working on a Python project that has external dependencies installed with pip, it is strongly recommended to first create a virtual environment.

Unfortunately that feature is easy to break: https://github.com/pypa/pip/issues/9644

I think I did my part https://github.com/pypa/pip/issues/6409

$ sudo pip install y$ sudo pip install yaml WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. ERROR: Could not find a version that satisfies the requirement yaml (from versions: none) ERROR: No matching distribution found for yaml

What are you implying will happen?

Using the build-in tools, you can save the exact versions of dependencies (i.e. a lock file) using "pip freeze >dependencies.txt". This should give you the exact same set of packages in two years' time.

If you want to be even more sure, you can also store hashes in the lock file. This has to be generated by a separate tools at the moment [1][2] but can be consumed by the built-in tools [3], so "pip install -r requirements.txt" is still all you need in two years' time.

[1] https://github.com/pypa/pip/issues/4732

[2] https://pip-tools.readthedocs.io/en/latest/#using-hashes

[3] https://pip.pypa.io/en/stable/topics/secure-installs/#hash-c...

Here’s the issue, https://github.com/pypa/pip/issues/11664. I think the idea would be to have some file/json description of environment that could be passed to pip to allow it to fully cross compile. They are open to supporting it just needs contributor to be found to implement it and go through review/discussion.

To be fair the only alternative is fixing Python, and even then you still would have to wait a good 5 years at least for all the old Python versions to dwindle.

It doesn't look like the fixing effort is progressing very quickly: https://github.com/pypa/pip/issues/8606

To their credit, at least they didn't close it "works as intended" which I imagine a lot of projects would.

Another good benchmark to trying to resolve apache-airflow[all]==1.10.13 using the state of PyPi on 2020-12-02, I give instructions here on how to reproduce that workflow: https://github.com/pypa/pip/issues/11836. Including a benchmark how how many extra packages your resolver should visit.

The main issue most people have with asdf is that it’s annoyingly slow. Not unusably so, but just enough that it’s irritating.

I identified [0] the source for much of it (sub-shells and pipes) and began a PR [1], but became bogged down with BATS testing, and then found mise / rtx, so kind of lost interest. Sorry. You can always implement these if you’d like.

[0]: https://github.com/asdf-vm/asdf/issues/290#issuecomment-1383...

[1]: https://github.com/asdf-vm/asdf/pull/1441

Or if you need to manage more than just node, asdf has been around for over a decade and works great. You can use a .tool-versions to change runtimes for each project you have, in addition to managing your global runtime versions

https://asdf-vm.com/

Why not just use a tool like asdf (https://asdf-vm.com/) or mise (https://mise.jdx.dev/)?

These tools have the advantage of not being multi-taskers and can manage version for all your tools. You wouldn’t need pyenv and npm and rvm and…

We’ve even started committing the .mise.toml files for projects to our repos. That way, since we work on multiple projects that may need multiple versions of the same tool, it’s handled and documented.

The purpose of a version manager is to help you navigate or install any tools for development easily. Version Manager can be one tool for each dependency (e.g. NVM, g) or One tool for all dependencies (e.g. asdf, mise).

(asdf)[https://asdf-vm.com/] fully supports Python and almost any other language. I've been using it for Ruby, Python, Elixir, and other languages for years and never looked back.

Secondly, our development environments must not drift, because then code may behave differently and a change could pass on our machine but fail in production. There are many tools for locking down environments, e.g nix, pkgx, asdf, containers, etc., and they all share the common goal of being able to lock down dependencies for an environment accurately and deterministically. And that needs to be enforced in our local workflow so we don't have to rely on CI environments for correctness. All developers must have environments that are effectively identical to what runs in CI (which itself should be representative of the production environment).

There are many ways this can be done (e.g nix, pkgx, asdf, containers, etc.), and we won’t get into which specific tools to use, because we'll instead cover the essential essence of preventing environment drift:

I've really been enjoying asdf, which is a program that allows you to install specified versions of dev utilities as well as dynamically manage them via shims and .tool-versions files.