Airflow setup/environment and best practices

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • cert-manager

    Automatically provision and manage TLS certificates in Kubernetes

  • Try to use mTLS as much as possible. Certmanager might help: https://cert-manager.io/

  • secrets-store-csi-driver-provider-aws

    The AWS provider for the Secrets Store CSI Driver allows you to fetch secrets from AWS Secrets Manager and AWS Systems Manager Parameter Store, and mount them into Kubernetes pods.

  • For a secrets manager we use the aws secrets store csi driver to fetch our secrets from aws secrets manager and parameter store. On Azure we still need to implement something similar, however an implementation does exist we haven't gotten around to it yet ;)

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts