Luigi Open-Source Projects

luigi

14 17,372 6.3 Python

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Project mention: Ask HN: What is the correct way to deal with pipelines? | news.ycombinator.com | 2023-09-21

I agree there are many options in this space. Two others to consider:
- https://airflow.apache.org/
- https://github.com/spotify/luigi
There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory…

InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Luigi related posts

In the context of Python what is a Bob Job?

2 projects | /r/learnpython | 10 Jul 2022
Apache Airflow. How to make the complex workflow as an easy job

1 project | dev.to | 20 Feb 2022
Open Source ETL Project For Startups

3 projects | dev.to | 22 Sep 2021
Resources/tutorials to help me learn about ETL?

1 project | /r/dataengineering | 29 Jun 2021
How to best integrate make-like functionality with invoke tasks/library?

1 project | /r/Python | 22 Mar 2021
Creating a scalable Flask Application

1 project | /r/flask | 26 Jan 2021
Spotify/luigi: build complex pipelines of batch jobs in Python

1 project | news.ycombinator.com | 15 Jan 2021
A note from our sponsor - SaaSHub
www.saashub.com | 21 May 2024

SaaSHub helps you find the best software and product alternatives Learn more →