squire
spark-style-guide
| squire | spark-style-guide | |
|---|---|---|
| 1 | 2 | |
| 24 | 250 | |
| - | - | |
| 10.0 | 0.0 | |
| over 1 year ago | 12 months ago | |
| Go | Jupyter Notebook | |
| MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
squire
-
Ask HN: How do you test SQL?
On a related note (though it does touch upon testing) mitchellh open sourced Squire[1] recently (today?). Note though, that it's PostgreSQL specific.
[1] https://github.com/mitchellh/squire
spark-style-guide
-
Ask HN: How do you test SQL?
Spark makes it easy to wrap SQL in functions that are easy to test. I am the author of the popular Scala Spark (spark-fast-tests) and PySpark (chispa) testing libraries. Some additional tips to speed up Spark tests (can speed up tests between 70-90%):
* reuse the same Spark session throughout the test suite
* Set shuffle partitions to 2 (instead of default which is 200)
* Use dependency injection to avoid disk I/O in the test suite
* Use fast DataFrame equality when possible. assertSmallDataFrameEquality is 4x faster than assertLargeDataFrameEquality.
* Use column equality to test column functions. Don't compare DataFrames unless you're testing custom DataFrame transformations. See the spark-style-guide for definitions for these terms: https://github.com/MrPowers/spark-style-guide/blob/main/PYSP...
Spark is an underrated tool for testing SQL. Spark makes it really easy to abstract SQL into unit testable chunks. Configuring your tests properly takes some knowledge, but you can make the tests run relatively quickly.
-
PySpark style guide
I created a PySpark style guide to help the community write code that's easy to reuse, unit test, and debug. Feel free to open issues / PRs if you have any suggestions / improvements.
What are some alternatives?
SS-Unit - A 100% T-SQL based unit testing framework for SQL Server
dbt-unit-testing - This dbt package contains macros to support unit testing that can be (re)used across dbt projects.
sqlx - 🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite.
pg_temp - create a temporary, disposable, userland pg database
pg-mem - An in memory postgres DB instance for your unit tests
pyspark-style-guide - This is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos we've encountered.
hash-db - Experimental distributed pseudomultimodel keyvalue database (it uses python dictionaries) imitating dynamodb querying with join only SQL support, distributed joins and simple Cypher graph support and document storage
datajudge - Assessing whether data from database complies with reference information.
bytebase - The GitLab/GitHub for database DevOps. World's most advanced database DevOps and CI/CD for Developer, DBA and Platform Engineering teams.
data-diff - Compare tables within or across databases
integresql - IntegreSQL manages isolated PostgreSQL databases for your integration tests.