Anyone's org using Airflow as a generalized job orchestator, not just for data engineering/ETL?

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering
Workflow Machine Learning Data Science data-engineering Mlops

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • ploomber

    The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

    • I can talk about the open-source I'm working on Ploomber (https://github.com/ploomber/ploomber), it's focusing on seamless integration with Jupyter and IDEs. It allows an easy mechanism to orchestrate work for instance, here's an example SQL ETL and then you can deploy it anywhere, so if you're working with Airflow, it'll deploy it there too but without the complexity. You wouldn't have to maintain docker images etc.

  • projects

    Sample projects using Ploomber. (by ploomber)

    • I can talk about the open-source I'm working on Ploomber (https://github.com/ploomber/ploomber), it's focusing on seamless integration with Jupyter and IDEs. It allows an easy mechanism to orchestrate work for instance, here's an example SQL ETL and then you can deploy it anywhere, so if you're working with Airflow, it'll deploy it there too but without the complexity. You wouldn't have to maintain docker images etc.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • New to large SW projects in Python, best practices to organize code

    1 project | /r/Python | 11 Nov 2022

  • A three-part series on deploying a Data Science Platform on AWS

    1 project | /r/dataengineering | 4 Nov 2022

  • Is Colab still the place to go?

    1 project | /r/deeplearning | 2 Nov 2022

  • Saving log files

    1 project | /r/docker | 26 Oct 2022

  • Teaching a senior chemical engineer python, should I teach them just python or python through jupyter?

    1 project | /r/Python | 20 Oct 2022