chispa VS meltano

here's a little README fix a user pushed to chispa

chispa is a library of PySpark testing functions.

The other approach I've used is Poetry, see the chispa project as an example. Poetry is especially nice for projects that you'd like to publish to PyPi because those commands are built-in.

- All Spark transformations are tested with pytest + chispa (https://github.com/MrPowers/chispa)

Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.

I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.

Blogging has been the main way I've been able to attract users. Someone searches "testing PySpark", they see this blog, and then they're motivated to try chispa.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

The last point was for teams that only rely on notebooks, sorry if I didn't make that clear.

You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files.

Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6

I would use something like meltano or airbyte, but if you really want to use Lambda for extraction I'd say there is no point spinning up a Redshift cluster just for that, Athena would be the way to go and you can use dbt pretty nicely with it and it would keep costs down.

Airbyte and Singer/Meltano if you want to learn more about ingestion pipelines. Airbyte and Meltano teams are very welcoming. SQLfluff a shiny SQL linter. Beautiful project with awesome maintainers.

I know really sure if this is what are you looking for, but take a look at Meltano

I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges.

[0] https://gitlab.com/meltano/meltano/-/issues/2616

At the same time, we have been working with some members of the community on a new framework for building taps and targets: https://gitlab.com/meltano/meltano/-/issues/2401, which we have decided to call the Singer SDK: https://gitlab.com/meltano/singer-sdk. We are moving as many Singer specification-specific details around things like incremental state replication and stream/field selection into the framework, so that individual taps only need to worry about getting the data from the source and can be expected to behave more consistently and correctly across the board.

Good point! As expected, there's an issue about adding it already: https://gitlab.com/meltano/meltano/-/issues/1175

I've fixed it: https://gitlab.com/meltano/meltano/-/merge_requests/2023. Should go live shortly.

Lastly, we recently put up a list of Implementation Partners on the Meltano website, which so far lists 7 consultancies that have adopted the Meltano+Singer standard, including multi-nationals with significant engineering resources dedicated to internal tools. A few of them had found Singer before Meltano and built their own tooling around Singer, but decided after finding Meltano to join the party and collaborate with us on better tooling and connectors we can all benefit from. The upcoming Singer SDK (background) is an example of such a collaboration with Slalom.

Meltano is open source, self-hosted, CLI-first, debuggable, and extensible.

At GitLab, we're not ready to give up on the Singer spec, community, and ecosystem yet, which is why I've been working on Meltano for the past year: https://meltano.com/

We think that the biggest things holding back Singer are the lack of documentation and tooling around taking existing taps and targets to production, and around building, debugging, maintaining, and testing new or existing high-quality taps and targets.

Meltano itself addresses the first problem, and provides a robust and reliable platform for building, running & orchestrating Singer- and dbt-based ELT pipelines.

At the same time, we have been working with some members of the community on a new framework for building taps and targets: https://gitlab.com/meltano/meltano/-/issues/2401, which we have decided to call the Singer SDK: https://gitlab.com/meltano/singer-sdk