datajudge
spark-style-guide
| datajudge | spark-style-guide | |
|---|---|---|
| 1 | 2 | |
| 38 | 250 | |
| - | - | |
| 8.1 | 0.0 | |
| 29 days ago | 11 months ago | |
| Python | Jupyter Notebook | |
| BSD 3-clause "New" or "Revised" License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
datajudge
-
Ask HN: How do you test SQL?
https://github.com/QuantCo/datajudge
We've also written a blog post trying to illustrate a use case:
spark-style-guide
-
Ask HN: How do you test SQL?
Spark makes it easy to wrap SQL in functions that are easy to test. I am the author of the popular Scala Spark (spark-fast-tests) and PySpark (chispa) testing libraries. Some additional tips to speed up Spark tests (can speed up tests between 70-90%):
* reuse the same Spark session throughout the test suite
* Set shuffle partitions to 2 (instead of default which is 200)
* Use dependency injection to avoid disk I/O in the test suite
* Use fast DataFrame equality when possible. assertSmallDataFrameEquality is 4x faster than assertLargeDataFrameEquality.
* Use column equality to test column functions. Don't compare DataFrames unless you're testing custom DataFrame transformations. See the spark-style-guide for definitions for these terms: https://github.com/MrPowers/spark-style-guide/blob/main/PYSP...
Spark is an underrated tool for testing SQL. Spark makes it really easy to abstract SQL into unit testable chunks. Configuring your tests properly takes some knowledge, but you can make the tests run relatively quickly.
-
PySpark style guide
I created a PySpark style guide to help the community write code that's easy to reuse, unit test, and debug. Feel free to open issues / PRs if you have any suggestions / improvements.
What are some alternatives?
SS-Unit - A 100% T-SQL based unit testing framework for SQL Server
dbt-unit-testing - This dbt package contains macros to support unit testing that can be (re)used across dbt projects.
pg_temp - create a temporary, disposable, userland pg database
fugue - A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
pyspark-style-guide - This is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos we've encountered.
sqlx - 🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite.
data-diff - Compare tables within or across databases
testcontainers-dotnet - A library to support tests with throwaway instances of Docker containers for all compatible .NET Standard versions.
integresql - IntegreSQL manages isolated PostgreSQL databases for your integration tests.
Flyway - Flyway by Redgate • Database Migrations Made Easy.
sqlfluff - A modular SQL linter and auto-formatter with support for multiple dialects and templated code.