chispa VS snipp.in

Unit and e2e tests using a combination of pytest and chispa (https://github.com/MrPowers/chispa). Custom library to create random test data that fits schema with optional hardcoded overrides for relevant fields to test business logic.

here's a little README fix a user pushed to chispa

chispa is a library of PySpark testing functions.

The other approach I've used is Poetry, see the chispa project as an example. Poetry is especially nice for projects that you'd like to publish to PyPi because those commands are built-in.

- All Spark transformations are tested with pytest + chispa (https://github.com/MrPowers/chispa)

Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.

I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.

Blogging has been the main way I've been able to attract users. Someone searches "testing PySpark", they see this blog, and then they're motivated to try chispa.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

I built https://snipp.in because I was top frustrated about the loading times of notion. I am a developer and I take notes quite often, most of us spend a lot of time on an IDE and when we need format a quick json or copy a tiny snippet we often find ourselves in opening another instance of an editor with a txt file or random website to formate json. Snipp.in is a tiny jn browser editor, note taking app and a snippet manager all in one which looks like an Editor but really fast without any bs of signin/signups or cloud. It just stores everything in your browser using IndexedDB. It's also open-sourced at https://github.com/haxzie/snipp.in