debuglater
mpire
debuglater | mpire | |
---|---|---|
8 | 8 | |
52 | 1,910 | |
- | 1.5% | |
3.8 | 7.5 | |
3 months ago | 12 days ago | |
Python | Python | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
debuglater
-
How do you deal with parallelising parts of an ML pipeline especially on Python?
Finally, debugging. If you're running code in sub-processes; debugging becomes a real pain because out of the box, you won't be able to start a debugger in the sub-processes. Furthermore, there's a chance that more than one fails. One solution is to dump the traceback when any sub-process fails, so you can start a debugging sesstion afterward; look at this project for an example.
-
debuglater: dump Python traceback for later debugging
The implementation is quite interesting. You can see it here. The serialization step has two parts: it takes the traceback object and wraps it into a new object so it can be serialized; secondly, it stores the source code so you can debug even if you don't have access to the source code!
-
debuglater: store Python traceback for later debugging
You can see a quick video demo here.
- GitHub - ploomber/debuglater: Store Python traceback for later debugging.
-
Show HN: Debuglater – Serialize Python traceback for later debugging
Hi HN!
We just released debuglater (https://github.com/ploomber/debuglater), an open-source library that serializes a Python traceback object for later debugging.
You can see a quick video demo here: https://github.com/ploomber/debuglater/blob/master/README.md
Countless times, we've scheduled overnight jobs to find out the following day that they failed. While logs are helpful, they are often insufficient for debugging. debuglater allows you to store the traceback object so you can start a debugging session at any moment.
We built this to support our open-source framework for data scientists (https://github.com/ploomber/ploomber), who often execute long-running code in remote environments. However, we realized this could be useful for the Python community, so we created a separate package. This project is a fork of Eli Finer's pydump, so kudos to him for laying the foundations!
The implementation is quite interesting. You can see it here (https://github.com/ploomber/debuglater/blob/master/src/debug...). The serialization step has two parts: it takes the traceback object and wraps it into a new object so it can be serialized; secondly, it stores the source code so you can debug even if you don't have access to the source code!
Please take it for a spin and let us know your feedback. Please share your feedback!
-
debuglater: Serialize Python traceback for later debugging
We just released debuglater, an open-source library that serializes a Python traceback object for later debugging.
- debuglater: Store Python traceback for later debugging. 🐛
mpire
- GitHub - sybrenjansen/mpire: A Python package for easy multiprocessing, but faster than multiprocessing
- Mpire: A Python package for easier and faster multiprocessing
-
Which not so well known Python packages do you like to use on a regular basis and why?
mpire for multiprocessing.
-
How do you deal with parallelising parts of an ML pipeline especially on Python?
https://github.com/Slimmer-AI/mpire is a nice lib, with better performance than multiprocessing.
-
Dask – a flexible library for parallel computing in Python
Shout out to an alternative to Dask: MPIRE https://github.com/Slimmer-AI/mpire
- Multi-Threading in Python
-
I'd like to introduce MPIRE: MultiProcessing Is Really Easy
After several iterations of feedback and exposure to production environments, it is now the go-to multiprocessing library at Slimmer AI. Recently, we’ve made it publicly available on GitHub (https://github.com/Slimmer-AI/mpire).
What are some alternatives?
ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
Dask - Parallel computing with task scheduling
pout - Python pretty print on steroids
cudf - cuDF - GPU DataFrame Library
pystack - 🔍 🐍 Like pstack but for Python!
distributed - A distributed task scheduler for Dask
pathml - Tools for computational pathology
cunumeric - An Aspiring Drop-In Replacement for NumPy at Scale
legate.pandas - An Aspiring Drop-In Replacement for Pandas at Scale
pyroute2 - Python Netlink and PF_ROUTE library — network configuration and monitoring
best-of-python - 🏆 A ranked list of awesome Python open-source libraries and tools. Updated weekly.
orchest - Build data pipelines, the easy way 🛠️