Spark: local dev environment

This page summarizes the projects mentioned and recommended in the original post on

Our great sponsors
  • Zigi - The context switching struggle is real
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • Scout APM - Truly a developer’s best friend
  • SonarLint - Clean code begins in your IDE with SonarLint
  • Apache Spark

    Apache Spark - A unified analytics engine for large-scale data processing

    The authors of Spark itself do a great job, poke around source for examples of tests:

  • chispa

    PySpark test helper methods with beautiful error messages

    - All Spark transformations are tested with pytest + chispa (

  • Zigi

    The context switching struggle is real. Zigi makes context switching a thing of the past. It monitors Jira and GitHub updates, pings you when PRs need approval and lets you take fast actions - all directly from Slack!

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts