Poetry VS pyflow

You can manage dependencies in Python with the package manager pip, which comes pre-installed with Python. Pip allows you to install and uninstall Python packages, and it uses a requirements.txt file to keep track of which packages your project depends on. However, pip does not have robust dependency resolution features or isolate dependencies for different projects; this is where tools like pipenv and poetry come in. These tools create a virtual environment for each project, separating the project's dependencies from the system-wide Python environment and other projects.

Poetry provides packaging and dependency management for Python. If you haven't already, install poetry via pip:

Poetry

The Semantify repository provides an example Astro.js project. Ensure you have poetry installed, then build the project from the root of the repository:

Has anyone else been paying attention to how hilariously hard it is to package PyTorch in poetry?

https://github.com/python-poetry/poetry/issues/6409

Based on this comment 5 days ago[0], it's working? I'm not sure didn't dig in too far but based on that comment it seems fair to say that it's not fully Poetry's fault because torch removed hashes (which poetry needs to be effective) for a while only recently adding it back in.

Not sure where I would stand if I fully investigated it tho.

[0] https://github.com/python-poetry/poetry/issues/6409#issuecom...

We will be running this project in Python 3.10 on Mac/Linux, and we will use Poetry to manage our dependencies. Later, we will bundle our app into a container using docker for deployment.

Here are the two main packaging issues I run into, specifically when using Poetry:

1) Lack of support for building extension modules (as mentioned by the article). There is a workaround using an undocumented feature [0], which I've tried, but ultimately decided it was not the right approach. I still use Poetry, but build the extension as a separate step in CI, rather than kludging it into Poetry.

2) Lack of support for offline installs [1], e.g. being able to download the dependencies, copy them to another machine, and perform the install from the downloaded dependencies (similar to using "pip --no-index --find-links=."). Again, you can work around this (by using "poetry export --with-credentials" and "pip download" for fetching the dependencies, then firing up pypiserver [2] to run a local PyPI server on the offline machine), but ideally this would all be a first class feature of Poetry, similar to how it is in pip.

I don't have the capacity to create Pull Requests for addressing these issues with Poetry, and I'm very grateful for the maintainers and those who do contribute. Instead, on the linked issues I share my notes on the matter, in the hope that it may at least help others and potentially get us closer to a solution.

Regardless, I'm sticking with Poetry for now. Though to be fair, the only other Python packaging tools I've used extensively are Pipenv and pip/setuptools. It's time consuming to thoroughly try out these other packaging tools, and is generally lower priority than developing features/fixing bugs, so it's helpful to read about the author's experience with these other tools, such as PDM and Hatch.

[0] https://github.com/python-poetry/poetry/issues/2740

[1] https://github.com/python-poetry/poetry/issues/2184

[2] https://pypi.org/project/pypiserver/

We believe that poetry is currently the best tool for this purpose, besides of being the most popular one at the moment. This is why we will use poetry to manage the dependencies of our project throughout this series of posts. Poetry allows you to declare the libraries your project depends on, and it will manage (install/update) them for you. Poetry also allows you to package your project into a distributable format and publish it to a repository, such as PyPI. We strongly recommend you to learn more about this tool by reading the official documentation.

I started using poetry. The problem is poetry will not install if there is dependency conflict and there is no way to ignore: github

Very cool! Of note, I made something along these lines a few years ago, although with a slightly broader scope to also include managing and installing python versions. I abandoned it due to lack of free time, and edge cases breaking things. The major challenge is that Python packages that aren't wheels can do surprising things due to setup.py running arbitrary code. (https://github.com/David-OConnor/pyflow)

Pyflow

Pyflow takes care of the use when you need pyenv to isolate different python versions, pipx to isolate some global python-based tools, and isolated, reproducible builds per project with on tool. I highly recommend people to give it go.

https://github.com/David-OConnor/pyflow#a-thoroughly-biased-...

Pyflow is a similar implementation of PEP582. NGL I wonder if it's better because of how good Rust stuff is. Probably a lot faster. Looks like you can install it via Pypi. I should've tested it before moving to PDM. Though it seems dev is a bit slow. Hmmm.

I’m personally complaining that pip is so much behind cargo. I have some hope with Pyflow though.

I literally stumbled into this issue again today. Has anyone leveraged Pyflow before? It looks pretty slick for keeping things organized. I don't do heavy dev work, just need something to keep things generally tidy. Was curious if anyone had used it and their opinion on it.

PDM is pretty new so it’s not entirely clear how it’ll play out but if you’re interested in PEP 582 then it’s really that or pyflow.

It's a good safeguard, and it's going in the direction of the other initiatives to make python package management default behavior saner.

PEP 852 is the another one to follow up: https://www.python.org/dev/peps/pep-0582/

It basically uses the concept of node_modules, making python interpreters local any local __pypackages__ directory. There are 2 differences though:

- unlike JS, python can only have one version of one lib

- but since having several versions of python often matters, you may have several __pypackages__/X.Y sub dirs to catter to each of them

It does also force you to use "-m" to use commands, which is the best practice anyway. I hope it will make jupyter fix "-m" on windows for them because that's a blocker for beginners.

If you are not already using "-m", start now. It solves a lot of different problems with running python cli programs.

E.G: instead of running "black" or "pylint", do "python -m black" or "python -m pylint". Or course you may want to chose a specific version of python, so "python3.8 -m black" for unix, or "py -3.8 -m black" on windows.

To test out __pypackages__, give a try to the pdm project: https://github.com/pdm-project/pdm

At last, some other tools that I wish people knew more about that solves packaging issues:

- pyflow (https://github.com/David-OConnor/pyflow): it's a package manager like poetry, but it also install whatever python you want like pyenv. Except it provides the binary, no need to compile anything. It's a young project, but I wish it succeeds because it's really a great concept.

- shiv (shiv.readthedocs.io/): it leverage the concept of zipapp, meaning the ability that python has to execute python inside a zip file. It's a successor to pex. Basically it lets you bundle your code + all deps from virtualenv inside a zip, like a Java .war file. You can then run the resulting zip, a .pyz file, like if it was a regular .py file. It will unzip on the first run automatically. It makes deployment almost as easy as with golang.

- nuitka (shiv.readthedocs.io/): take your code and all dependancies, turn them into C, and compiles it. Although it does require a bit of setup, since it needs headers and a compiler, it results reliably in a standalone compiled executable that will run on the same architecture with no need for anything else. Also it will speed up your Python program, up to 4 times.