Top 12 Python data-profiling Projects

• ydata-profiling

  43 12,022 8.5 Python

  1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

  

Project mention: FLaNK 25 December 2023 | dev.to | 2023-12-26

• great_expectations

  15 9,440 9.9 Python

  Always know what to expect from your data.

  

Project mention: Data Quality at Scale with Great Expectations, Spark, and Airflow on EMR | dev.to | 2023-04-24

Great Expectations (GE) is an open-source data validation tool that helps ensure data quality.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• cleanlab

  69 8,592 9.4 Python

  The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

  

Project mention: [Research] Detecting Annotation Errors in Semantic Segmentation Data | /r/MachineLearning | 2023-11-05

We have feely open-sourced our new method for improving segmentation data, published a paper on the research behind it, and released a 5-min code tutorial. You can also read more in the blog if you'd like.

• sweetviz

  1 2,833 6.7 Python

  Visualize and compare datasets, target values and associations, with one line of code.

• soda-core

  5 1,745 9.0 Python

  :zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io

  

• Optimus

  0 1,441 1.9 Python

  :truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark (by ironmussa)

  

• cleanvision

  4 919 7.7 Python

  Automatically find issues in image datasets and practice data-centric computer vision.

  

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• popmon

  1 485 6.9 Python

  Monitor the stability of a Pandas or Spark dataframe ⚙︎

  

• piperider

  6 466 9.5 Python

  Code review for data in dbt

  

Project mention: Show HN: PipeRider – open-source Data Impact Analysis for dbt changes | news.ycombinator.com | 2023-09-06

• swiple

  1 77 2.7 Python

  Swiple enables you to easily observe, understand, validate and improve the quality of your data

  

• data-profiling

  1 67 4.9 Python

  a set of scripts to pull meta data and data profiling metrics from relational database systems

• metacrafter

  1 38 3.7 Python

  Metadata and data identification tool and Python library. Identifies PII, common identifiers, language specific identifiers. Fully customizable and flexible rules

  

Project mention: Metacrafter – semantic data types detection Python lib | news.ycombinator.com | 2024-03-13

Python data-profiling related posts

• Show HN: PipeRider – open-source Data Impact Analysis for dbt changes
  3 projects | news.ycombinator.com | 6 Sep 2023
• Looking for Unit Testing framework in Database Migration Process
  3 projects | /r/dataengineering | 23 Mar 2023
• Data profiling as part of a data reliability strategy?
  2 projects | /r/dataengineering | 15 Sep 2022
• Show HN: PipeRider, data reliability automated tool
  2 projects | news.ycombinator.com | 23 Jun 2022

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!