What should I learn to improve as a data engineer?

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/dataengineering

Our great sponsors
  • Scout APM - Truly a developer’s best friend
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • SonarQube - Static code analysis for 29 languages.
  • talent.io - Download talent.io’s Tech Salary Report
  • helm

    The Kubernetes Package Manager

    For K8s, we were using Cloud Composer to do it for us but wanted more fine control over CI/CD, so we decided to go with Airflow on K8s. That's all hosted in GKE now and deployed using ArgoCD with helm. This also led down the IaC rabbit hole which has been a ton of fun too. We use the GCP ConfigConnector resources for that which is a little challenging at first, but gets a lot easier as time goes on.

  • k8s-config-connector

    GCP Config Connector, a Kubernetes add-on for managing GCP resources

    For K8s, we were using Cloud Composer to do it for us but wanted more fine control over CI/CD, so we decided to go with Airflow on K8s. That's all hosted in GKE now and deployed using ArgoCD with helm. This also led down the IaC rabbit hole which has been a ton of fun too. We use the GCP ConfigConnector resources for that which is a little challenging at first, but gets a lot easier as time goes on.

  • Scout APM

    Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts