Unit testing your PySpark library

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • OPS - Build and Run Open Source Unikernels
  • Scout APM - Less time debugging, more time building
  • SonarQube - Static code analysis for 29 languages.
  • example-pyspark-unittesting

    An example of unit testing PySpark code

    If you want to have a look at the code, or use it in any way (it's MIT licensed), then you can find it over on Github.

  • Apache Spark

    Apache Spark - A unified analytics engine for large-scale data processing

    In software development we often unit test our code (hopefully). And code written for Spark is no different. So here I want to run through an example of building a small library using PySpark and unit testing it. I'm using Visual Studio Code as my editor here, mostly because I think it's brilliant, but other editors are available.

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • Visual Studio Code

    Visual Studio Code

    In software development we often unit test our code (hopefully). And code written for Spark is no different. So here I want to run through an example of building a small library using PySpark and unit testing it. I'm using Visual Studio Code as my editor here, mostly because I think it's brilliant, but other editors are available.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts