quinn VS yadm

We're brainstorming functions to make PySpark easier, see this issue: https://github.com/MrPowers/quinn/issues/83

Adding some README documentation to the README should be quite straightforward. Here's a function that needs to be documented: https://github.com/MrPowers/quinn/issues/52 .

quinn is a library with PySpark helper functions. I need to work through all the open issues / PRs and bump all versions. I should do another release. This library gets around 600,000 monthly downloads.

Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.

Here's how I added a create_df method to the SparkSession class: https://github.com/MrPowers/quinn/blob/main/quinn/extensions/spark_session_ext.py

They spent a lot of effort improving the catalyst engine under the hood too and making it easier to extend and improve it in the future. Making it easy to add your own native code to Spark itself. Shameless plug of a blog post I wrote on this subject which basically reiterates what Matthew Powers, author of Spark Daria and quinn, wrote here.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

Everyone hand-rolls their own dotfile management system, but YADM already does everything you need:

https://yadm.io/

I've been working around this using tools built on top of git like [yadm](https://github.com/TheLocehiliosan/yadm) and relying on `ls-files` to list all my tracked dotfiles and their paths.

Still having everything in one place would make things much simpler. Great idea!

I wonder if the program i use to manage my dotfiles could help manage your scripts and extend your setup to all your desktops? Its called yadm (https://yadm.io/) it makes it so easy to have a laptop and a desktop or two.

I really like that one but still prefer yadm because you can just edit your files as usual and then yadm add them wherever you are.

If you haven’t already, this is the time to install a tool like yadm and get your computer configuration into version control. Your command-line tools can be managed by yadm directly, your system settings can mostly be managed with a yadm bootstrap script that runs things like defaults write, and the software you install can be managed with a Brewfile that the yadm bootstrap script uses to install software with Homebrew. Don’t manually download Xcode, use xcodes to do it.

I personally use YADM. It's basically a git repo on my home folder, that only tracks what I explicitly set. And you can setup bootstraps to do what you said, install a bunch of stuff or make custom changes. In it's essence, it's a set of bash/sh files that are executed sequentially when you launch the yadm bootstrap command.