chispa VS meltano

Unit and e2e tests using a combination of pytest and chispa (https://github.com/MrPowers/chispa). Custom library to create random test data that fits schema with optional hardcoded overrides for relevant fields to test business logic.

here's a little README fix a user pushed to chispa

chispa is a library of PySpark testing functions.

The other approach I've used is Poetry, see the chispa project as an example. Poetry is especially nice for projects that you'd like to publish to PyPi because those commands are built-in.

- All Spark transformations are tested with pytest + chispa (https://github.com/MrPowers/chispa)

Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.

I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.

Blogging has been the main way I've been able to attract users. Someone searches "testing PySpark", they see this blog, and then they're motivated to try chispa.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

- https://github.com/meltano/meltano

We'd love to hear what you think of Meltano (Cloud). If you join the Beta, you get 100 free credits (200 hourly or 100 daily runs) and a 20% discount on the pricing at GA (June 27). The first 100 to sign up get 1,000 credits -- that's 83 days of hourly runs or 3 years of dailies!

The team and I will be checking in here throughout the day, so don't hesitate to ask questions! If we don't get to you, feel free to join 3,500+ Meltano fans on https://meltano.com/slack and we'll chat there!

Meltano[0] might be of interest to you. Easy way to move data that should be very familiar for software engineers. If a connector doesn't exist our SDK makes it easy to build it.

[0] https://github.com/meltano/meltano

(disclaimer - I work at Meltano)

We currently don't do any process optimization on a per-stream basis when doing an extract. We have seen folks in the community running each tap separately for each stream which can speed it up. We've got an issue around this (Melturbo).

Meltano

I built my data pipeline to Extract some data from websites and CSV files, Load it into my database, and Transform it into a reporting-ready schema. I used Python and Pandas to extract and load some of the data and Meltano to load some additional supporting data. All of that data went into a PostgreSQL database hosted in the cloud on Azure where I then used dbt to create data models in the database optimized for reporting. Finally, I use Metabase to visualize the data. (whew! that's a lot of moving parts!)

I'm learning https://meltano.com/ right now, so am building custom Taps, mostly for fun. I'm enjoying it. I'm pulling in a variety of data from https://www.geonames.org/ and Canadian weather/climate data into BigQuery

https://meltano.com/ is ELT but I like it