SaaSHub helps you find the best software and product alternatives Learn more →
Top 19 data-cleaning Open-Source Projects
-
cleanlab
The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
-
miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
-
-
Optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark (by ironmussa)
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
Encord Active
Open source active learning toolkit to find failure modes in your computer vision models, prioritize data to label next, and drive data curation to improve model performance.
-
-
desbordante-core
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
-
-
-
-
quantclean
🧹 Quantclean is a program that reformats financial dataset to US Equity TradeBar (Quantconnect format)
-
csv_log_cleaner
Clean CSV files to conform to a type schema by streaming them through small memory buffers using multiple threads and logging data loss.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: [Research] Detecting Annotation Errors in Semantic Segmentation Data | /r/MachineLearning | 2023-11-05We have feely open-sourced our new method for improving segmentation data, published a paper on the research behind it, and released a 5-min code tutorial. You can also read more in the blog if you'd like.
Project mention: Are there any Python libraries for Data Cleansing ? | /r/dataengineering | 2023-12-08
Project mention: Launch HN: Encord (YC W21) – Unit testing for computer vision models | news.ycombinator.com | 2024-01-31We base our pricing on your user and consumption scale and would be happy to discuss this with you directly. Please feel free to explore the OS version of Active at https://github.com/encord-team/encord-active. Note that some features, such as natural language search using GPU accelerated APIs, are not included in the cloud version.
Project mention: FuzzTypes: Pydantic Library for Auto-Correcting Annotation Types | news.ycombinator.com | 2024-03-15
Week 0: 🌩️ Image Quality Issues & 📈 Concept Interpolation
Among all these feel-good stories, how about one with a bit different ending?
During my masters, I created a ML library that dealt with noise in dataset. I implemented bunch of papers, but unlike your usual research code, I spent a long time obsessing about it's API, performance, created documentation, CI- the whole shebang [1]. But then, like avg research code, I moved on and promptly forgot about it.
One day about a year ago the cofounder of a very new, small startup working on something similar texted me about the project on linkedin. We chatted for a bit, but as a guy who thinks he's too cool for linkedin, I next logged in and saw his last message about wanting to collaborate about 3/4 months after the fact.
Well they raised $25 million dollars a few months ago :(
[1] https://github.com/Shihab-Shahriar/scikit-clean
Week 4: 🪞Image Deduplication
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
data-cleaning related posts
-
Show HN: Desbordante 1.0.0 Released
-
[Research] Detecting Annotation Errors in Semantic Segmentation Data
-
[R] Automated Quality Assurance for Object Detection Datasets
-
Perl first commit: a “replacement” for Awk and sed
-
Need help on cleaning this data!!
-
Running weekly average
-
GQL: A new SQL like query language for .git files written in Rust
-
A note from our sponsor - SaaSHub
www.saashub.com | 10 May 2024
Index
What are some of the best open-source data-cleaning projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | cleanlab | 8,719 |
| 2 | miller | 8,571 |
| 3 | Mage | 7,085 |
| 4 | pandera | 3,029 |
| 5 | Optimus | 1,446 |
| 6 | janitor | 1,342 |
| 7 | skrub | 1,012 |
| 8 | schema-inspector | 504 |
| 9 | Encord Active | 420 |
| 10 | validate | 402 |
| 11 | desbordante-core | 354 |
| 12 | feature-engineering-tutorials | 266 |
| 13 | FuzzTypes | 199 |
| 14 | akvo-lumen | 63 |
| 15 | image-quality-issues | 20 |
| 16 | quantclean | 17 |
| 17 | scikit-clean | 13 |
| 18 | image-deduplication-plugin | 8 |
| 19 | csv_log_cleaner | 2 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com