flintrock VS databricks-nutter-repos-demo

> * AWS has a managed Spark offering called EMR

There is also my rinky-dink open source project, Flintrock [0], that will launch open source Spark clusters on AWS for you.

It's probably not the right tool for production use (and you would be right to wonder why Flintrock exists when we have EMR [1]), but I know of several companies that have used Flintrock at one point or other in production at large scale (like, 400+ node clusters).

[0]: https://github.com/nchammas/flintrock

[1]: https://github.com/nchammas/flintrock#why-build-flintrock-wh...

You can use Databricks Repos (https://docs.databricks.com/repos/index.html) specifically files in repos (https://docs.databricks.com/repos/work-with-notebooks-other-...) functionality that allows to use Python files (not notebooks!) as Python modules.

Another alternative is to split notebooks into “library notebooks” that just define transformations, and “orchestration notebooks” that use code library notebooks to execute a “business logic”.

In both approaches you can do code testing, etc.

P.S. I have a demo of both approaches here: https://github.com/alexott/databricks-nutter-repos-demo

I’m sorry for delay, will fix ASAP...

My point is that you can do that even without jars/wheels - you can do VC and tests of notebooks. For example, https://github.com/alexott/databricks-nutter-projects-demo