pyenv VS pip-tools

For more information visit: pyenv repository

Note that virtual environments assume you are using the same global version of Python. Often, this is not the case and additional tools like pyenv can be used alongside virtual environments when you need to switch between versions of Python itself on your local machine.

Python version manager pyenv

First of all, install the Python virtual environment from these links: 1 and 2. I developed my GPT-based API in Python version 3.8.18. Pick any Python versions >= 3.7.

Note: Most modern systems will probably have a system environment that meets this requirement, but if yours does not or if you prefer not to install anything in your system environment (even if it's just PDM) check out asdf or pyenv to help install and manage additional Python environments.

When dealing with software development, reproducibility is key. This is why we encourage you to use Python virtual environments to set up an isolated environment for your project. Virtual environments allow the isolation of dependencies, which plays a crucial role to avoid breaking compatibility between different projects. We cannot cover all the details about virtual environments in this post, but we encourage you to learn more about venv, pyenv or conda for a better understanding on how to create and manage virtual environments.

For Python, with your use case I would avoid system packages, no matter the distro. It sounds like it would be worth setting up pyenv and working exclusively with virtual environments.

For OSX there is homebrew or pyenv (pyenv is another solution on Linux). As pyenv compiles from source it will require setting up XCode (the Apple IDE) tools to support this which can be pretty bulky. Windows users have chocolatey but the issue there is it works off the binaries. That means it won't have the latest security release available since those are source only. Conda is also another solution which can be picked up by Visual Studio Code as available versions of Python making development easier. In the end it might be best to consider using WSL on Windows for installing a Linux version and using that instead.

> Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts.

My friend, here is what you seek: https://github.com/jazzband/pip-tools

requirements.txt is flat because it's really the output of `pip freeze`. It's supposed to completely and exactly rebuild the environment. Unfortunately it's far too flexible and people abuse it by putting in only direct dependencies etc.

If you're writing packages, you don't need a requirements.txt at all, by the way. Package dependencies (only direct dependencies) live in pyproject.toml with the rest of the package config. requirements.txt (and pip tools) are only for when you want to freeze the whole environment, like for a server deployment.

For all my projects I found myself regenerating manual lock files using complex shell commands with pip-compile to get a reproducible environments across devices using a custom pre-install-command. I finally decided that instead of hacking together the same solution on all my projects I would build a plugin that handles this complexity for me.

Instead of venv, we are using pip-tools in this starter kit. pip-tools take things further in dependency management. Check out what pip-tools does in their official GitHub repo. In short, it helps your project find the best match for the dependent packages. For example, you might need two packages A and B in your project that requires same package C under the hood. But A requires any version of C from 1.0.1 to 1.0.10 and B requires any version of C from 1.0.7 to 1.0.15. Pip tools will automatically compile the version of 'C' that suits for both of your packages.

I've created a small project called just-pip-tools that combines pip-tools and just to manage Python dependencies in a layered approach. This isn't a magic bullet; it's a set of files you can adapt to your needs.

For small projects I recommend pip-tools. Just write packet list in requirements.in and pip-compile compile a requirements.txt with comments.

Take a look at pip-tools, great package. https://github.com/jazzband/pip-tools

pip-tools

I recommend generating a lockfile to document this information, as you might do with pip-tools.

Check out pip-tools [1] which does exactly that, albeit in a slightly more polished way.

[1]: https://github.com/jazzband/pip-tools