data-diff

Compare tables within or across databases (by datafold)

Data-diff Alternatives

Similar projects and alternatives to data-diff

  1. sqlx

    151 data-diff VS sqlx

    🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite. (by launchbadge)

  2. CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
  3. graphjin

    GraphJin - Build NodeJS / GO APIs in 5 minutes not weeks

  4. orchest

    44 data-diff VS orchest

    Build data pipelines, the easy way 🛠️

  5. bytebase

    World's most advanced database DevSecOps solution for Developer, Security, DBA and Platform Engineering teams. The GitHub/GitLab for database DevSecOps.

  6. sqlfluff

    35 data-diff VS sqlfluff

    A modular SQL linter and auto-formatter with support for multiple dialects and templated code.

  7. Lark

    35 data-diff VS Lark

    Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.

  8. objectiv-analytics

    Discontinued Open-source product analytics infrastructure for data teams that want full control. Built for high quality data collection and ready to use for advanced analytics & ML.

  9. Nutrient

    Nutrient - The #1 PDF SDK Library. Bad PDFs = bad UX. Slow load times, broken annotations, clunky UX frustrates users. Nutrient’s PDF SDKs gives seamless document experiences, fast rendering, annotations, real-time collaboration, 100+ features. Used by 10K+ devs, serving ~half a billion users worldwide. Explore the SDK for free.

    Nutrient logo
  10. testcontainers-dotnet

    A library to support tests with throwaway instances of Docker containers for all compatible .NET Standard versions.

  11. Preql

    16 data-diff VS Preql

    An interpreted relational query language that compiles to SQL.

  12. great_expectations

    Always know what to expect from your data.

  13. pg-mem

    14 data-diff VS pg-mem

    An in memory postgres DB instance for your unit tests

  14. cuallee

    5 data-diff VS cuallee

    Possibly the fastest DataFrame-agnostic quality check library in town.

  15. datacompy

    Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

  16. soda-core

    5 data-diff VS soda-core

    :zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io

  17. fugue

    11 data-diff VS fugue

    A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

  18. dbhub.io

    A "Cloud" for SQLite databases. Collaborative development for your data. 😊

  19. prism

    7 data-diff VS prism

    Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python. (by runprism)

  20. dbt-snowflake-monitoring

    A dbt package from SELECT to help you monitor Snowflake performance and costs

  21. integresql

    IntegreSQL manages isolated PostgreSQL databases for your integration tests.

  22. hash-db

    Discontinued Experimental distributed pseudomultimodel keyvalue database (it uses python dictionaries) imitating dynamodb querying with join only SQL support, distributed joins and simple Cypher graph support and document storage [GET https://api.github.com/repos/samsquire/hash-db: 404 - Not Found // See: https://docs.github.com/rest/repos/repos#get-a-repository]

  23. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better data-diff alternative or higher similarity.

data-diff discussion

Log in or Post with

data-diff reviews and mentions

Posts with mentions or reviews of data-diff. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-07-26.
  • How to Check 2 SQL Tables Are the Same
    8 projects | news.ycombinator.com | 26 Jul 2023
    If the issue happen a lot, there is also: https://github.com/datafold/data-diff

    That is a nice tool to do it cross database as well.

    I think it's based on checksum method.

  • Oops, I wrote yet another SQLAlchemy alternative (looking for contributors!)
    4 projects | /r/pythoncoding | 8 May 2023
    First, let me introduce myself. My name is Erez. You may know some of the Python libraries I wrote in the past: Lark, Preql and Data-diff.
  • Looking for Unit Testing framework in Database Migration Process
    3 projects | /r/dataengineering | 23 Mar 2023
    https://github.com/datafold/data-diff might be worth a look
  • Ask HN: How do you test SQL?
    18 projects | news.ycombinator.com | 31 Jan 2023
    I did data engineering for 6 years and am building a company to automate SQL validation for dbt users.

    First, by “testing SQL pipelines”, I assume you mean testing changes to SQL code as part of the development workflow? (vs. monitoring pipelines in production for failures / anomalies).

    If so:

    1 – assertions. dbt comes with a solid built-in testing framework [1] for expressing assertions such as “this column should have values in the list [A,B,C]” as well checking referential integrity, uniqueness, nulls, etc. There are more advanced packages on top of dbt tests [2]. The problem with assertion testing in general though is that for a moderately complex data pipeline, it’s infeasible to achieve test coverage that would cover most possible failure scenarios.

    2 – data diff: for every change to SQL, know exactly how the code change affects the output data by comparing the data in dev/staging (built off the dev branch code) with the data in production (built off the main branch). We built an open-source tool for that: https://github.com/datafold/data-diff, and we are adding an integration with dbt soon which will make diffing as part of dbt development workflow one command away [2]

    We make money by selling a Cloud solution for teams that integrates data diff into Github/Gitlab CI and automatically diffs every pull request to tell you the how a change to SQL affects the target table you changed, downstream tables and dependent BI tools (video demo: [3])

    I’ve also written about why reliable change management is so important for data engineering and what are key best practices to implement [4]

    [1] https://docs.getdbt.com/docs/build/tests

  • Data-diff v0.3: DuckDB, efficient in-database diffing and more
    1 project | news.ycombinator.com | 15 Dec 2022
    Hi HN:

    We at Datafold are excited to announce a new release of data-diff (https://github.com/datafold/data-diff), an open-source tool that efficiently compares tables within or across a wide range of SQL databases. This release includes a lot of new features, improvements and bugfixes.

    We released the first version 6 months ago because we believe that diffing data is as fundamental of a capability as diffing code in data engineering workflows. Over the past few months, we have seen data-diff being adopted for a variety of use-cases, such as validating migration and replication of data between databases (diffing source and target) and tracking the effects of code changes on data (diffing staging/dev and production environments).

    With this new release data-diff is significantly faster at comparing tables within the same database, especially when there are a lot of differences between the tables. We've also added the ability to materialize the diff results into a database table, in addition to (or instead of) outputting them to stdout. We've added support for DuckDB, and for diffing schemas. Improved support for alphanumerics, and threading, and generally improved the API, the command-line interface, and stability of the tool.

    We believe that data-diff is a valuable addition to the open source community, and we are committed to continue growing it and the community around it. We encourage you to try it out and let us know what you think!

    You can read more about data-diff on our GitHub page at the following link: https://github.com/datafold/data-diff/

    To see the list of changes for the 0.3.0 release, go here: https://github.com/datafold/data-diff/releases/tag/v0.3.0

  • data-diff VS cuallee - a user suggested alternative
    2 projects | 30 Nov 2022
  • Compare identical tables across databases to identify data differences (Oracle 19c)
    1 project | /r/SQL | 26 Oct 2022
  • How to test Data Ingestion Pipeline
    1 project | /r/dataengineering | 26 Sep 2022
    For data mismatches, check out data-diff https://github.com/datafold/data-diff
  • Data migration - easier way to compare legacy with new environment?
    1 project | /r/dataengineering | 6 Sep 2022
  • Show HN: Open-source infra for building embedded data pipelines
    2 projects | news.ycombinator.com | 1 Sep 2022
    Looks useful! Do you have a way to validate that the data was copied correctly and entirely? If not, you might want to consider integrating data-diff for that - https://github.com/datafold/data-diff
  • A note from our sponsor - CodeRabbit
    coderabbit.ai | 14 Feb 2025
    Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR. Learn more →

Stats

Basic data-diff repo stats
20
2,899
9.4
9 months ago

Sponsored
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai

Did you know that Python is
the 2nd most popular programming language
based on number of references?