keda VS karpenter-provider-aws

It seems you want something like KEDA (https://keda.sh)

Most just utilize out of the box macro resources available in HPA.

For more advanced use cases there is keda - https://keda.sh/

Thankfully KEDA operator was already part of the cluster, and all Robin had to do was create a ScaledObject manifest targeting the Dispatch ScaleUp event, based on the rabbitmq_global_messages_received_total metric from Prometheus.

Keda

KEDA introduces event-driven scaling to Kubernetes workloads. It integrates with Kubernetes Horizontal Pod Autoscalers and can scale pods based on external metrics from services like databases and message queues (Kafka, RabbitMQ, MongoDB).

# update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml

This is traditionally a AWS Lambda usecase - or an OpenFaas kind of usecase. But very recently i discovered https://keda.sh/ and it seems it is specifically meant for this in a kubernetes environment.

If you deploy the application to Amazon EKS, you can also consider using KEDA to auto-scale your consumer application based on the number of messages in the MSK topic.

If you’re not familiar with Karpenter — watch my quick intro. But in a nutshell, Karpenter is a better node autoscaler for Kubernetes (say goodbye to wasted compute resources). It is open-source and built by the AWS team. Qovery is an Internal Developer Platform I’m a co-founder) that we’ll use to spin up our EKS cluster with Karpenter.

Karpenter

Here are a few reference links about the previous services and tools: What is Amazon EKS? Cluster Mesh Karpenter

Autoscaling over EKS can be accomplished using either the cluster-autoscaler project or Karpenter. If you want to use Spot instances, consider using Karpenter, as it has better integrations with AWS for optimizing spot pricing and availability, minimizing interruptions, and falling back to on-demand nodes if no spot instances are available.

Kubernetes has its own learning curve, but when tools like Karpenter exist it's kinda hard to beat for "auto-scaled compute" that is vendor agnostic. We leverage Karpenter for burst in our vSphere environment as well as our EC2 environment. Karpenter is invoking roughly the same Terraform code in both cases, just using different modules for the particular virtualization. Say we want to go to Azure and GCP -- we add an Azure and GCP module to the same Terraform codebase, and not much else needs to change from the "scale up / scale down" perspective.

Also https://github.com/aws/karpenter/issues/331

Why don't you stick to the KubernetesPodOperator though? I fail to see a benefit in using the ECS operator considering you're already running Airflow in EKS. You can look into something like karpenter to manage your nodes.

We use the Kubernetes runner and Karpenter.