spark-fast-tests VS Nullboard

I was a Databricks user for 5 years and spent 95% of my time developing Spark code in IDEs. See the spark-daria and spark-fast-tests projects as Scala examples. I developed internal libraries with all the business logic. The Databricks notebooks would consist of a few lines of code that would invoke a function in the proprietary Spark codebase. The proprietary Spark codebase would depend on the OSS libraries I developed in parallel.

https://github.com/MrPowers/spark-fast-tests https://github.com/97arushisharma/Scala_Practice/tree/master/BigData_Analysis_with_Scala_and_Spark/wikipedia

If the majority of your stuff is not UDF-based there is an OS solution to run assertion tests against full data frames called spark-fast-tests. The idea here is similar in that you have a it notebook that calls your actual notebook against a staged input reads the output and compares it to a prefabed expected output. This does take a bit of setup and trial and error but it’s the closest I’ve been able to get to proper automated regression testing in databricks

I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

Hello, HN! I'd like to introduce you to nbagent[0], an open-source alternative backup agent for Nullboard[1] for Unix systems. If you, like me, are a fan of Nullboard's simplicity and you've been looking for an easy way to have your data backed up, check it out.

I'd love to hear your feedback on nbagent. Please feel free to leave comments, suggestions, or questions below.

[0] https://github.com/luismedel/nbagent

[1] https://nullboard.io/preview

Surprisingly no one has mentioned nullboard (https://nullboard.io/preview) yet. Just a single file JSON to carry around the entire board.

Nullboard maybe?

minimalist kanban board, without fancy tagging/time tracking/... features. Every game has a column, and notes are player names. nullboard would be perfect if it wasn't local only;

Nullboard - cool simple project, no bug tracking

Nullboard - Kanban board with dark mode, localStorage, and thoughtful UI

I use and like https://nullboard.io/preview nullboard which runs in the browser. Because it uses local data, you need to save it or the data is lost when you close the browser window.