mpire
DataProfiler
mpire | DataProfiler | |
---|---|---|
8 | 61 | |
1,910 | 1,363 | |
1.5% | 1.0% | |
7.5 | 6.3 | |
9 days ago | 6 days ago | |
Python | Python | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mpire
- GitHub - sybrenjansen/mpire: A Python package for easy multiprocessing, but faster than multiprocessing
- Mpire: A Python package for easier and faster multiprocessing
-
Which not so well known Python packages do you like to use on a regular basis and why?
mpire for multiprocessing.
-
How do you deal with parallelising parts of an ML pipeline especially on Python?
https://github.com/Slimmer-AI/mpire is a nice lib, with better performance than multiprocessing.
-
Dask โ a flexible library for parallel computing in Python
Shout out to an alternative to Dask: MPIRE https://github.com/Slimmer-AI/mpire
- Multi-Threading in Python
-
I'd like to introduce MPIRE: MultiProcessing Is Really Easy
After several iterations of feedback and exposure to production environments, it is now the go-to multiprocessing library at Slimmer AI. Recently, weโve made it publicly available on GitHub (https://github.com/Slimmer-AI/mpire).
DataProfiler
-
LongRoPE: Extending LLM Context Window Beyond 2M Tokens
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
- Data Profiler โ What's in your data?
-
Data Profiler 0.9.0 -- offering a massive improvement to memory usage during profiling of large datasets
Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Release 0.8.3 ยท capitalone/DataProfiler
What are some alternatives?
Dask - Parallel computing with task scheduling
ydata-profiling - 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
cudf - cuDF - GPU DataFrame Library
pyWhat - ๐ธ Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! ๐งโโ๏ธ
distributed - A distributed task scheduler for Dask
usaddress - :us: a python library for parsing unstructured United States address strings into address components
pathml - Tools for computational pathology
XlsxWriter - A Python module for creating Excel XLSX files.
cunumeric - An Aspiring Drop-In Replacement for NumPy at Scale
superset - Apache Superset is a Data Visualization and Data Exploration Platform
legate.pandas - An Aspiring Drop-In Replacement for Pandas at Scale
vtuber-livechat-dataset - ๐ VTuber 1B: Billion-scale Live Chat and Moderation Event Dataset