Data pipelines with Luigi

• luigi

  14 17,310 6.4 Python

  Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  

At Wonderflow we're doing a lot of ML / NLP using Python and recently we are enjoying writing data pipelines using Spotify's Luigi.

• spaCy

  106 28,704 9.2 Python

  💫 Industrial-strength Natural Language Processing (NLP) in Python

  

We have tasks which actually require lots of different Spacy language models to be loaded at once, and we load them on many processes at once.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Dask

  32 11,999 9.6 Python

  Parallel computing with task scheduling

  

To do that, we are efficiently using Dask, simply creating on-demand local (or remote) clusters on task run() method:

• Airflow

  169 34,485 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Moreover, configure and deploy the Luigi's Scheduler on a server / pod for production use is easy, while it might be not for other similar tools like Apache AirFlow.

Suggest a related project