Python Packages Project Generator VS projects

- python-package-template

Hey HN, this would be more of an early release (still planning on some tweaks before a release) -- would love to hear your thoughts on this!

For some back-story, this is more of a side-side-project (made this while working on another side-project).

When I switched to using Go for my projects (from Python), the lack of a template generator similar to python-package-template[1] was very annoying. I would copy the basic files (Makefile, Github actions, PR templates, etc) from the previous project only to realize I forgot to change some stuff, and now would need to rewrite git history.

By the third project, I decided to create a template generator for Go! I've tried to keep the generated project as flexible as possible - you can decide to skip the of it and go for a simple project, or take the bloat (pre-commit would need Python for one).

While making go-template, one of my side goals has been to keep the project beginner-friendly. I remember stumbling upon python-package-template[1] as a novice, and learning more than I had in a semester - Makefiles, linters, code-formatters, semantic versioning, pipelines, and so much more! With go-template, I hope to give that same experience to some other newbie who might stumble upon my repo (or a project generated using go-template).

As a fun fact, go-template has an option to remove Github-specific-features (pull request templates, workflows, etc). This was inspired by a comment on HN[2] pointing out that many open-source projects were on Github simply because of FOMO, which in-turn promoted Github's dominance!

[1]: https://github.com/TezRomacH/python-package-template

Oh, and if I am not mistaken, you have also used the python-package-template itself to generate goli structure 🔥

CookieCutter or Kedro are the winners. I still think we will stick to Kedro template, because it offers extra functionality, and I like to think of each project as a set of pipelines to be run. Anyway, some cookiecutter templates are very good, like this one. In case we use both Kedro and ClearML, we'll have to figure out how to integrate its pipelines with ClearML tasks. But in the slack channel of ClearML there are other teams doing the same, so at least it's possible.

You can look at the files in detail here. For this tutorial, I'll quickly mention a few crucial details.

Ploomber is the complete solution for notebook execution. It builds on top of papermill and extends it to allow writing multi-stage workflows where each task is a notebook. Meanwhile, it automatically manages orchestration. Hence you can run notebooks in parallel without having to write extra code.

The answer is YES, you can take advantage of OOP best practices to write good ETLs. For instance in this Ploomber sample ETL You can see there's a mix of .sql and .py files, it's within modular components so it's easier to test, deploy and execute. It's way easier than airflow since there's no infra work involved, you only have to setup your pipeline.yaml file. This also allows you to make the code WAY more maintainable and scalable, avoid redundant code and deploy faster :)

We have tons of examples that follow a standard layout, here’s one: https://github.com/ploomber/projects/tree/master/templates/ml-intermediate

I can talk about the open-source I'm working on Ploomber (https://github.com/ploomber/ploomber), it's focusing on seamless integration with Jupyter and IDEs. It allows an easy mechanism to orchestrate work for instance, here's an example SQL ETL and then you can deploy it anywhere, so if you're working with Airflow, it'll deploy it there too but without the complexity. You wouldn't have to maintain docker images etc.

I recommend using Ploomber which can help you build once and automate a lot of the work, and it works with python natively. It's open source so you can start with one of the examples, like the ML-basic example or the ETL one. It'll allow you to define the pipeline and then easily explain the flow with the DAG plot. Feel free to ask questions, I'm happy to help (I've built 100s of data pipelines over the years).

I'm not sure what pipeline frameworks support this kind of testing, but after successfully implementing this workflow, I added this feature to Ploomber, the project I'm working on. Here's how a pipeline looks like, and here's a tutorial.

Check out Ploomber, (disclaimer: I'm the author) it has a simple API, and you can export to Airflow, AWS, Kubernetes. Supports all databases that work with Python and you can seamlessly transfer from a SQL step to a Python step. Here's an example.

Without more specifics about your use case, it's hard to give more specific advice. But check out Ploomber (disclaimer: I'm the creator) - here's an example ETL pipeline. I've used it in past projects to develop Oracle ETL pipelines. Modularizing the analysis in many parts helps a lot with maintenance.

Yes! (tell your friend). You can write shell scripts so you can execute that 2002 code :) You can test it locally and then run it in AWS Batch/Argo. Here's an example