For those using Airflow for your ELT/Orchestration, How are you perfroming your EL?

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering
data-pipelines dagster Workflow Data Science workflow-automation

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • analytics

    • On the data team at GitLab we're running Airflow in k8s. We use Stitch and Fivetran (and Meltano!) for a bunch of our EL but also trigger custom Python jobs via Airflow. Our repo is https://gitlab.com/gitlab-data/analytics/ if you're curious about the specifics.

  • dagster

    An orchestration platform for the development, production, and observation of data assets.

    • (T) : https://github.com/fishtown-analytics/dbt + https://github.com/great-expectations/great_expectations + https://github.com/dagster-io/dagster

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts