data-diff VS dbhub.io

If the issue happen a lot, there is also: https://github.com/datafold/data-diff

That is a nice tool to do it cross database as well.

I think it's based on checksum method.

First, let me introduce myself. My name is Erez. You may know some of the Python libraries I wrote in the past: Lark, Preql and Data-diff.

https://github.com/datafold/data-diff might be worth a look

I did data engineering for 6 years and am building a company to automate SQL validation for dbt users.

First, by “testing SQL pipelines”, I assume you mean testing changes to SQL code as part of the development workflow? (vs. monitoring pipelines in production for failures / anomalies).

If so:

1 – assertions. dbt comes with a solid built-in testing framework [1] for expressing assertions such as “this column should have values in the list [A,B,C]” as well checking referential integrity, uniqueness, nulls, etc. There are more advanced packages on top of dbt tests [2]. The problem with assertion testing in general though is that for a moderately complex data pipeline, it’s infeasible to achieve test coverage that would cover most possible failure scenarios.

2 – data diff: for every change to SQL, know exactly how the code change affects the output data by comparing the data in dev/staging (built off the dev branch code) with the data in production (built off the main branch). We built an open-source tool for that: https://github.com/datafold/data-diff, and we are adding an integration with dbt soon which will make diffing as part of dbt development workflow one command away [2]

We make money by selling a Cloud solution for teams that integrates data diff into Github/Gitlab CI and automatically diffs every pull request to tell you the how a change to SQL affects the target table you changed, downstream tables and dependent BI tools (video demo: [3])

I’ve also written about why reliable change management is so important for data engineering and what are key best practices to implement [4]

[1] https://docs.getdbt.com/docs/build/tests

Hi HN:

We at Datafold are excited to announce a new release of data-diff (https://github.com/datafold/data-diff), an open-source tool that efficiently compares tables within or across a wide range of SQL databases. This release includes a lot of new features, improvements and bugfixes.

We released the first version 6 months ago because we believe that diffing data is as fundamental of a capability as diffing code in data engineering workflows. Over the past few months, we have seen data-diff being adopted for a variety of use-cases, such as validating migration and replication of data between databases (diffing source and target) and tracking the effects of code changes on data (diffing staging/dev and production environments).

With this new release data-diff is significantly faster at comparing tables within the same database, especially when there are a lot of differences between the tables. We've also added the ability to materialize the diff results into a database table, in addition to (or instead of) outputting them to stdout. We've added support for DuckDB, and for diffing schemas. Improved support for alphanumerics, and threading, and generally improved the API, the command-line interface, and stability of the tool.

We believe that data-diff is a valuable addition to the open source community, and we are committed to continue growing it and the community around it. We encourage you to try it out and let us know what you think!

You can read more about data-diff on our GitHub page at the following link: https://github.com/datafold/data-diff/

To see the list of changes for the 0.3.0 release, go here: https://github.com/datafold/data-diff/releases/tag/v0.3.0

For data mismatches, check out data-diff https://github.com/datafold/data-diff

Looks useful! Do you have a way to validate that the data was copied correctly and entirely? If not, you might want to consider integrating data-diff for that - https://github.com/datafold/data-diff

We have a spread of different GitHub Actions based workflows that do stuff whenever a PR is proposed or merged:

https://github.com/sqlitebrowser/sqlitebrowser/tree/master/....

Most of those are oriented around building packages for various OS's (Linux, macOS, Windows) so people can try the latest code.

While there are some tests, they're more like extremely basic sanity tests and don't rely on Docker.

Those tests rely on whichever version of SQLite was downloaded and compiled into the GUI (as per above code snippet).

---

That being said, that's for the client side GUI application. There's a server side of things too (https://github.com/sqlitebrowser/dbhub.io -> dbhub.io) that does use docker for it's automated tests:

https://github.com/sqlitebrowser/dbhub.io/tree/master/.githu...

Those are integration tests though (eg "make sure we didn't bust communication with our cli", "make sure our go library still works 100% with the server"), and a reasonably decent set of End to End (E2E) tests of the web interface using Cypress.

---

Does that help? :)

> I now have some Go websites running for a long time with daily Systemd restarts.

Are the restarts needed because there's malicious over-size content being sent to it, trying to exhaust the server resources?

If so, then "http.MaxBytesReader" might be helpful:

https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...

Not sure if Go code is your kind of thing, but if it is my colleague wrote a SQLite "diff" capability for our online hosting operation:

https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...

The code there can also output a "merge" object out of the differences too, in order to merge the differences from one database object into another.

https://dbhub.io/ ?

It's open source, free, cross platform and frequently updated. You can also use https://dbhub.io as a way to backup and share your databases.

I just finished my project 'pydbhub ' for accessing and using SQLite databases on DBHub.io.