spark-daria VS yadm

I was a Databricks user for 5 years and spent 95% of my time developing Spark code in IDEs. See the spark-daria and spark-fast-tests projects as Scala examples. I developed internal libraries with all the business logic. The Databricks notebooks would consist of a few lines of code that would invoke a function in the proprietary Spark codebase. The proprietary Spark codebase would depend on the OSS libraries I developed in parallel.

They spent a lot of effort improving the catalyst engine under the hood too and making it easier to extend and improve it in the future. Making it easy to add your own native code to Spark itself. Shameless plug of a blog post I wrote on this subject which basically reiterates what Matthew Powers, author of Spark Daria and quinn, wrote here.

I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.

quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.

Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.

Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8

I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

Everyone hand-rolls their own dotfile management system, but YADM already does everything you need:

https://yadm.io/

I've been working around this using tools built on top of git like [yadm](https://github.com/TheLocehiliosan/yadm) and relying on `ls-files` to list all my tracked dotfiles and their paths.

Still having everything in one place would make things much simpler. Great idea!

I wonder if the program i use to manage my dotfiles could help manage your scripts and extend your setup to all your desktops? Its called yadm (https://yadm.io/) it makes it so easy to have a laptop and a desktop or two.

I really like that one but still prefer yadm because you can just edit your files as usual and then yadm add them wherever you are.

If you haven’t already, this is the time to install a tool like yadm and get your computer configuration into version control. Your command-line tools can be managed by yadm directly, your system settings can mostly be managed with a yadm bootstrap script that runs things like defaults write, and the software you install can be managed with a Brewfile that the yadm bootstrap script uses to install software with Homebrew. Don’t manually download Xcode, use xcodes to do it.

I personally use YADM. It's basically a git repo on my home folder, that only tracks what I explicitly set. And you can setup bootstraps to do what you said, install a bunch of stuff or make custom changes. In it's essence, it's a set of bash/sh files that are executed sequentially when you launch the yadm bootstrap command.