How would you QA data before/after a migration?

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering
dataquality data-testing data-monitoring Spark data-quality

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • deequ

    Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

    • check out https://github.com/awslabs/deequ

  • soda-sql

    Discontinued Data profiling, testing, and monitoring for SQL accessible data.

    • We just released a new open source tool for testing SQL accessible data and have support for BigQuery: https://docs.soda.io/soda-sql/

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts