Apache Airflow. How to make the complex workflow as an easy job

This page summarizes the projects mentioned and recommended in the original post on dev.to
Workflow engine Python Luigi orchestration-framework Scheduling

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • luigi

    Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

    • It's good to know what Airflow is not the only one on the market. There are Dagster and Spotify Luigi and others. But they have different pros and cons, be sure that you did a good investigation on the market to choose the best suitable tool for your tasks.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Resources/tutorials to help me learn about ETL?

    1 project | /r/dataengineering | 29 Jun 2021

  • How to best integrate make-like functionality with invoke tasks/library?

    1 project | /r/Python | 22 Mar 2021

  • Creating a scalable Flask Application

    1 project | /r/flask | 26 Jan 2021

  • Spotify/luigi: build complex pipelines of batch jobs in Python

    1 project | news.ycombinator.com | 15 Jan 2021

  • In the context of Python what is a Bob Job?

    2 projects | /r/learnpython | 10 Jul 2022