cicd-templates VS nutter

That's what Databricks Labs have done in this example I happened to find very shortly after posting this question: https://github.com/databrickslabs/cicd-templates . They run with local pyspark and dbx for launching jobs instead

From my experience OO gives you much more flexibility in designing your pipeline but you're risking to make the project way more complicated. The worst example I have seen is the Nutter library (https://github.com/microsoft/nutter), which uses endless classes that are all nested in each other. I once had a bug when using it, and it was a huge pain in the ass to understand what's going on when the code is executed. It is a very good example of what can go wrong when you're overusing OO. However, in one project, I carefully created few classes, just out of curiosity, and I was very impressed how it helped me to organize/structure my code. A functions hase a clear dedicated use, but a good class is like a Swiss army knife with an solid set of functionalities. If you know how to use it in a smart way, you are likely to increase the quality of your code, but the contrary is also very likely, especially when the team members are not ready for it.

- https://github.com/microsoft/nutter