Unit testing your PySpark library

• example-pyspark-unittesting

  1 2 3.2 Python

  An example of unit testing PySpark code

If you want to have a look at the code, or use it in any way (it's MIT licensed), then you can find it over on Github.

• Apache Spark

  101 38,414 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

In software development we often unit test our code (hopefully). And code written for Spark is no different. So here I want to run through an example of building a small library using PySpark and unit testing it. I'm using Visual Studio Code as my editor here, mostly because I think it's brilliant, but other editors are available.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Visual Studio Code

  2,851 158,564 10.0 TypeScript

  Visual Studio Code

  

In software development we often unit test our code (hopefully). And code written for Spark is no different. So here I want to run through an example of building a small library using PySpark and unit testing it. I'm using Visual Studio Code as my editor here, mostly because I think it's brilliant, but other editors are available.

Suggest a related project