Which best practices do you follow to build robust & extensible ETL jobs?

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering
Command-line Tools Command-line Application Development CLI Click Python3

WorkOS
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • typer

    Typer, build great CLIs. Easy to code. Based on Python type hints.

    • Most computing tasks in airflow DAGs are KubernetesPodOperator containing a CLI (Python Typer). It allows us to pass arguments easily to run DAG manually if needed (the new UI to pass arguments to DAG in airflow 2.6 is really nice). Arguments allow us to replay DAG easily (change start / end dates for instance).

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts