quinn VS fugue

We're brainstorming functions to make PySpark easier, see this issue: https://github.com/MrPowers/quinn/issues/83

Adding some README documentation to the README should be quite straightforward. Here's a function that needs to be documented: https://github.com/MrPowers/quinn/issues/52 .

quinn is a library with PySpark helper functions. I need to work through all the open issues / PRs and bump all versions. I should do another release. This library gets around 600,000 monthly downloads.

Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.

Here's how I added a create_df method to the SparkSession class: https://github.com/MrPowers/quinn/blob/main/quinn/extensions/spark_session_ext.py

They spent a lot of effort improving the catalyst engine under the hood too and making it easier to extend and improve it in the future. Making it easy to add your own native code to Spark itself. Shameless plug of a blog post I wrote on this subject which basically reiterates what Matthew Powers, author of Spark Daria and quinn, wrote here.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

Please integrate it with Fugue.

https://github.com/fugue-project/fugue

Fugue is an interesting library in this space , though I haven’t tried it

https://github.com/fugue-project/fugue

A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark, Dask and Ray without any rewrites.

fugue — distributed computing done easy

This looks similar to fugue, am I right? How do they compare?

I am looking forward to how Substrait could help removing this friction. It aims to provide a standardised intermediate query language (lower level than SQL) to connect frontend user interfaces like SQL or data frame libraries with backend analytical computing engines. It is linked to the Arrow ecosystem. Something like Ibis or Fugue could become the front and DuckDB the backend engine.

There's dask-sql, but I think it is being abandoned for fugue-project. I'm actually excited for this project as it is trying to provide a backend agnostic solution, which would seem like a difficult, lofty goal. I wish them luck.