Top 11 Python data-cleaning Projects

• cleanlab

  69 8,592 9.4 Python

  The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

  

Project mention: [Research] Detecting Annotation Errors in Semantic Segmentation Data | /r/MachineLearning | 2023-11-05

We have feely open-sourced our new method for improving segmentation data, published a paper on the research behind it, and released a 5-min code tutorial. You can also read more in the blog if you'd like.

• Mage

  77 7,001 9.9 Python

  🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

  

Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• pandera

  7 2,994 8.9 Python

  A light-weight, flexible, and expressive statistical data testing library

  

• Optimus

  0 1,441 1.9 Python

  :truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark (by ironmussa)

  

• skrub

  1 1,009 8.9 Python

  Prepping tables for machine learning

  

Project mention: Are there any Python libraries for Data Cleansing ? | /r/dataengineering | 2023-12-08

• Encord Active

  6 420 9.1 Python

  Open source active learning toolkit to find failure modes in your computer vision models, prioritize data to label next, and drive data curation to improve model performance.

  

Project mention: Launch HN: Encord (YC W21) – Unit testing for computer vision models | news.ycombinator.com | 2024-01-31

We base our pricing on your user and consumption scale and would be happy to discuss this with you directly. Please feel free to explore the OS version of Active at https://github.com/encord-team/encord-active. Note that some features, such as natural language search using GPU accelerated APIs, are not included in the cloud version.

• FuzzTypes

  1 188 9.0 Python

  Pydantic extension for annotating autocorrecting fields.

  

Project mention: FuzzTypes: Pydantic Library for Auto-Correcting Annotation Types | news.ycombinator.com | 2024-03-15

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• image-quality-issues

  1 20 7.9 Python

  FiftyOne Plugin for finding common image quality issues

Project mention: Plugin for Building and Managing Plugins! | dev.to | 2024-02-09

Week 0: 🌩️ Image Quality Issues & 📈 Concept Interpolation

• quantclean

  1 16 0.0 Python

  🧹 Quantclean is a program that reformats financial dataset to US Equity TradeBar (Quantconnect format)

• scikit-clean

  1 13 2.9 Python

  A collection of algorithms for detecting and handling label noise

Project mention: Ask HN: What side projects landed you a job? | news.ycombinator.com | 2023-12-03

Among all these feel-good stories, how about one with a bit different ending?

During my masters, I created a ML library that dealt with noise in dataset. I implemented bunch of papers, but unlike your usual research code, I spent a long time obsessing about it's API, performance, created documentation, CI- the whole shebang [1]. But then, like avg research code, I moved on and promptly forgot about it.

One day about a year ago the cofounder of a very new, small startup working on something similar texted me about the project on linkedin. We chatted for a bit, but as a guy who thinks he's too cool for linkedin, I next logged in and saw his last message about wanting to collaborate about 3/4 months after the fact.

Well they raised $25 million dollars a few months ago :(

[1] https://github.com/Shihab-Shahriar/scikit-clean

• image-deduplication-plugin

  1 8 5.7 Python

  Remove exact and approximate duplicates from your dataset in FiftyOne!

Project mention: Plugin for Building and Managing Plugins! | dev.to | 2024-02-09

Week 4: 🪞Image Deduplication

Python data-cleaning related posts

• [Research] Detecting Annotation Errors in Semantic Segmentation Data
  1 project | /r/MachineLearning | 5 Nov 2023
• [R] Automated Quality Assurance for Object Detection Datasets
  1 project | /r/computervision | 28 Sep 2023
• Need Suggestion for transformation tool
  1 project | /r/dataengineering | 21 Dec 2022
• Updating dbt Cloud pricing to support long-term community growth 50$->100$
  1 project | /r/dataengineering | 16 Dec 2022
• Airflow for Data Ingestion
  1 project | /r/dataengineering | 29 Nov 2022
• ETL tool
  3 projects | /r/dataengineering | 24 Nov 2022
• Show HN: Mage, Fivetran alternative for ELT and data integrations
  2 projects | news.ycombinator.com | 22 Nov 2022
• A note from our sponsor - InfluxDB
  www.influxdata.com | 23 Apr 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!