what do you think about airflow?

• Airflow

  169 34,397 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

I think one of the main design problems I have with Airflow is the fact that it tends to tightly couple processing/transform code with data movement code which makes debugging tricky. The way I have solved this is by building a command line interface to all the processing code so I can debug the processing code outside of any airflow infrastructure (which can be painful to get running locally if one does not use Airflow Breeze.

• awesome-pipeline

  10 5,904 5.7

  A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin

I found this list of other "awesome pipelines" https://github.com/pditommaso/awesome-pipeline

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project