mpire
pathml
mpire | pathml | |
---|---|---|
8 | 2 | |
1,910 | 362 | |
1.5% | 2.5% | |
7.5 | 8.0 | |
9 days ago | 25 days ago | |
Python | Python | |
MIT License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mpire
- GitHub - sybrenjansen/mpire: A Python package for easy multiprocessing, but faster than multiprocessing
- Mpire: A Python package for easier and faster multiprocessing
-
Which not so well known Python packages do you like to use on a regular basis and why?
mpire for multiprocessing.
-
How do you deal with parallelising parts of an ML pipeline especially on Python?
https://github.com/Slimmer-AI/mpire is a nice lib, with better performance than multiprocessing.
-
Dask – a flexible library for parallel computing in Python
Shout out to an alternative to Dask: MPIRE https://github.com/Slimmer-AI/mpire
- Multi-Threading in Python
-
I'd like to introduce MPIRE: MultiProcessing Is Really Easy
After several iterations of feedback and exposure to production environments, it is now the go-to multiprocessing library at Slimmer AI. Recently, we’ve made it publicly available on GitHub (https://github.com/Slimmer-AI/mpire).
pathml
- Hilo Semanal de Consultas IT - Asesoría Técnica, Desarrollo Profesional y Aprendizaje
-
Dask – a flexible library for parallel computing in Python
We have been using dask to support our computational pathology workflows [1], where the images are so big that they cannot be loaded in memory, let alone analyzed (standard pathology whole slide images are ~1GB; some microscopy techniques generate images >1TB). We divide each image into a bunch of smaller tiles and process each tile independently. The dask.distributed scheduler lets us scale up by distributing the tile processing across a cluster.
Benefits of dask.distributed: easy to get up and running, and has support for spinning up clusters on lots of different computing platforms (local machines, HPC cluster, k8s, etc.)
One difficulty is optimizing performance - there are so many configuration details (job size, number of workers, worker resources, etc. etc.) that it's been hard to know what is best.
[1] https://github.com/Dana-Farber-AIOS/pathml
What are some alternatives?
Dask - Parallel computing with task scheduling
slideflow - Deep learning library for digital pathology, with both Tensorflow and PyTorch support.
cudf - cuDF - GPU DataFrame Library
Keras - Deep Learning for humans
distributed - A distributed task scheduler for Dask
pytorch-ssim - pytorch structural similarity (SSIM) loss
cunumeric - An Aspiring Drop-In Replacement for NumPy at Scale
legate.pandas - An Aspiring Drop-In Replacement for Pandas at Scale
pyroute2 - Python Netlink and PF_ROUTE library — network configuration and monitoring