Noobie who is trying to use K8s needs confirmation to know if this is the way or he is overestimating Kubernetes.

• luigi

  14 17,292 6.4 Python

  Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  

• Airflow

  169 34,397 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• pipeline

  51 8,276 9.7 Go

  A cloud-native Pipeline resource.

  

Apart from airflow, or luigi, if you want a YAML-driven approach with some http api triggers, take a look at tekton (http://tekton.dev) . It can define tasks (as containers) and pipelines of multiple tasks. Has a GUI monitor tool if desired as well.

Suggest a related project