community VS karpenter-provider-aws

Multiple container runtimes are supported, like conatinerd, cri-o, or other CRI compliant runtimes.

More recently, the Kubernetes SIG Network has been evolving the Gateway API to support service meshes.

https://github.com/kubernetes/kubernetes https://www.kubernetes.dev/resources/calendar/ https://github.com/kubernetes/community/blob/master/governance.md https://github.com/kubernetes/steering https://github.com/kubernetes/community/blob/master/sig-list.md

According to this doc it takes the average of CPU utilization of a pod (average across the last 1 minute) divided by the CPU requested by the pod. Then it computes the arithmetic mean of all the pods' CPU.

metrics-server has not supported kubectl top Resource Metrics API

With the Gateway API being a superset of the Ingress API, it might make sense to consolidate both. Thanks to the SIG Network community, Gateway API is still growing and will soon be production ready.

Projects in/around Kubernetes and the CNCF are generally where I spend what little time I can these days. Most communities are incredibly welcoming and provide timely feedback. But the problem space of "managing a cloud platform" can take several years to really wrap ones head around, setting aside focused topics via SIGs like networking, storage, observability, API design, etc.

Krew is a plugin manager maintained by the Kubernetes Special Interest Group (SIG) CLI community. Krew makes it easy to use kubectl plugins and helps you discover, install, and manage them on your machine. It is similar to tools like apt, dnf, or brew. Today, over 200 kubectl plugins are available on Krew - and that number is only increasing. Some projects are actively used and some get deprecated over time, but are still accessible via Krew.

[1] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=Last%20decade&var-metric=contributions [2] https://kubernetes.io/releases/release/ [3] https://github.com/kubernetes/community/blob/master/governance.md [4] https://github.com/kubernetes/community/blob/master/sig-list.md

If you’re not familiar with Karpenter — watch my quick intro. But in a nutshell, Karpenter is a better node autoscaler for Kubernetes (say goodbye to wasted compute resources). It is open-source and built by the AWS team. Qovery is an Internal Developer Platform I’m a co-founder) that we’ll use to spin up our EKS cluster with Karpenter.

Karpenter

Here are a few reference links about the previous services and tools: What is Amazon EKS? Cluster Mesh Karpenter

Autoscaling over EKS can be accomplished using either the cluster-autoscaler project or Karpenter. If you want to use Spot instances, consider using Karpenter, as it has better integrations with AWS for optimizing spot pricing and availability, minimizing interruptions, and falling back to on-demand nodes if no spot instances are available.

Kubernetes has its own learning curve, but when tools like Karpenter exist it's kinda hard to beat for "auto-scaled compute" that is vendor agnostic. We leverage Karpenter for burst in our vSphere environment as well as our EC2 environment. Karpenter is invoking roughly the same Terraform code in both cases, just using different modules for the particular virtualization. Say we want to go to Azure and GCP -- we add an Azure and GCP module to the same Terraform codebase, and not much else needs to change from the "scale up / scale down" perspective.

Also https://github.com/aws/karpenter/issues/331

Why don't you stick to the KubernetesPodOperator though? I fail to see a benefit in using the ECS operator considering you're already running Airflow in EKS. You can look into something like karpenter to manage your nodes.

We use the Kubernetes runner and Karpenter.