karpenter-provider-aws VS kaniko

If you’re not familiar with Karpenter — watch my quick intro. But in a nutshell, Karpenter is a better node autoscaler for Kubernetes (say goodbye to wasted compute resources). It is open-source and built by the AWS team. Qovery is an Internal Developer Platform I’m a co-founder) that we’ll use to spin up our EKS cluster with Karpenter.

Karpenter

Here are a few reference links about the previous services and tools: What is Amazon EKS? Cluster Mesh Karpenter

Autoscaling over EKS can be accomplished using either the cluster-autoscaler project or Karpenter. If you want to use Spot instances, consider using Karpenter, as it has better integrations with AWS for optimizing spot pricing and availability, minimizing interruptions, and falling back to on-demand nodes if no spot instances are available.

Kubernetes has its own learning curve, but when tools like Karpenter exist it's kinda hard to beat for "auto-scaled compute" that is vendor agnostic. We leverage Karpenter for burst in our vSphere environment as well as our EC2 environment. Karpenter is invoking roughly the same Terraform code in both cases, just using different modules for the particular virtualization. Say we want to go to Azure and GCP -- we add an Azure and GCP module to the same Terraform codebase, and not much else needs to change from the "scale up / scale down" perspective.

Also https://github.com/aws/karpenter/issues/331

Why don't you stick to the KubernetesPodOperator though? I fail to see a benefit in using the ECS operator considering you're already running Airflow in EKS. You can look into something like karpenter to manage your nodes.

If all you need to do is build container, you can use https://github.com/GoogleContainerTools/kaniko

Kaniko for building the container images

kaniko

You should maybe look into Kaniko or use some other build tool

If you are using the docker socket just for building container images, you might want to look into kaniko. It doesn't use docker to build images. If you use the socket also for starting containers (we are actually doing that in our CI pipelines), you could think about limiting the pods Kubernetes schedules on a node (you can change the default of 110 using the kubelet config file).

Check out Kaniko for building containers https://github.com/GoogleContainerTools/kaniko . Only issue is it doesnt support windows containers.

It took less than 5 minutes to install. It quickly analysed the repo and identified easy ways to make the project more secure. Priya Wadhwa, Kaniko

I'd suggest to take a look at the Kaniko project, combined with custom container images in Lambda functions.

kaniko is a tool that allows you to build container images inside Kubernetes without the need for the Docker daemon. Effectively, it allows you to build Docker images without docker build.

Kubernetes prefers to pull containers from registries. You may be able to work around it by specifying a local image in your Kube manifest. Both https://github.com/GoogleContainerTools/kaniko and/ or https://www.devspace.sh/ may help.