eksctl VS Kubernetes-Volume-Autoscaler

There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using eksctl CLI because of the convenience it offers. Creating an an EKS cluster using eksctl, can be as easy as this:

There are many ways to create a cluster such as using eksctl. In my case, I will use terraform module cause it’s easy to reuse and comprehend.

eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.

For this and many other reasons I recommend doing everything in Terraform EXCEPT EKS and its node groups. For that, I use https://eksctl.io/ because it much better manages the lifecycle of EKS and your node groups. I have an blog article better explaining why I recommend it, and another blog article explaining how to do zero-downtime upgrades with EKSCTL.

eksctl

Second, make sure you create a spot instance group that attempts to launch MULTIPLE different instance types. This way if one instance type gets flushed, your autoscaler will kick in and launch a different type. Without this, you WILL HAVE DOWNTIME if a sudden price hike and flush occurs. If you're using eksctl I have example configurations that use multi-instance types on Github here.

There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using eksctl CLI because of the convenience it offers. Creating an an EKS cluster using eksctl, can be as easy as this:

I've written an article on this, with my recommended tool for managing eks EKSCTL.

--- AWSTemplateFormatVersion: '2010-09-09' Parameters: VpcId: Type: AWS::EC2::VPC::Id Description: ID of the VPC in which to create the Kubernetes cluster SubnetIds: Type: List Description: List of Subnet IDs in which to create the Kubernetes cluster KeyPairName: Type: AWS::EC2::KeyPair::KeyName Description: Name of the EC2 Key Pair to use for SSH access to worker nodes ClusterName: Type: String Description: Name of the Kubernetes cluster to create Resources: ControlPlaneSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: VpcId: !Ref VpcId GroupDescription: Allow inbound traffic to the Kubernetes control plane SecurityGroupIngress: - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0 WorkerNodeSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: VpcId: !Ref VpcId GroupDescription: Allow inbound traffic to Kubernetes worker nodes SecurityGroupIngress: - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0 ControlPlaneInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Roles: - !Ref ControlPlaneRole ControlPlaneRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy - arn:aws:iam::aws:policy/AmazonEKSServicePolicy ControlPlaneInstance: Type: AWS::EC2::Instance Properties: ImageId: ami-0b69ea66ff7391e80 InstanceType: t2.micro KeyName: !Ref KeyPairName NetworkInterfaces: - DeviceIndex: 0 AssociatePublicIpAddress: true GroupSet: - !Ref ControlPlaneSecurityGroup SubnetId: !Select [0, !Ref SubnetIds] IamInstanceProfile: !Ref ControlPlaneInstanceProfile UserData: Fn::Base64: !Sub | #!/bin/bash echo 'net.bridge.bridge-nf-call-iptables=1' | tee -a /etc/sysctl.conf sysctl -p yum update -y amazon-linux-extras install docker -y service docker start usermod -a -G docker ec2-user curl -o /usr/local/bin/kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl chmod +x /usr/local/bin/kubectl echo 'export PATH=$PATH:/usr/local/bin' >> /etc/bashrc curl --silent --location "https://github.com/weaveworks/eksctl/releases

Additionally if you don't know, Kubernetes freshly setup, especially AWS's EKS is largely useless after you first set it up. You need to then install roughly a dozen other services into it to make it "do all the magic automatically". Services such as aws-ebs-csi-driver, (optional) aws-efs-csi-driver, (optional) aws-fsx-csi-driver, aws-load-balancer-controller, (optional) aws-node-termination-handler, cluster-autoscaler, (optional) external-dns, logs cascading engine (eg: fluentd-elasticsearch / fluent-bit-elasticsearch / datadog), grafana, prometheus, your ingress controller of choice (I prefer and recommend ingress-nginx), and the Kubernetes Volume Autoscaler to auto-scale up EBS volumes. (shameless plug: I wrote the volume-autoscaler)

Shameless Plug: Here's one of my dashboards I made for Ingress-Nginx, which is my recommended border router/gateway into all the services. It adds deep robust metrics and configurability, and if you've got years of experience with Nginx also, it allows you rich complex customization via nginx's configuration structure via kubernetes annotations. Besides that I have open-source helm charts which are easy to use, boilerplates showing how to use them, a volume autoscaler to automatically resize your disks as they get full, and a blog where I share various of my experience which is a companion blog to my upcoming book of the same name. Hope this helps! Feel free to ask if you have any further questions.

Do not waste your timesetting up your own Kubernetes cluster; use any cloud provider's fully managed Kubernetes cluster, and then learn how to configure everything on it to do everything you want. Typically, there are anywhere between 10-30 foundational services you'll want to install on it to make everything work. Things such as Cluster-Autoscaler, an ingress controller, a mesh network technology, various CSI volume provisioners, a runner for your chosen CI/CD platform, a disk volume autoscaler (shameless plug I wrote this) etc. Learn to deploy Helm charts on it, and learn to deploy some of your services onto it, exposing them to the internet. Learn to install and use Prometheus and Grafana on it to get in-depth metrics and visualization. Learn how to use Prometheus Alertmanager to trigger alerts to your email, webhooks, slack, etc. There's a lot to learn, and it may feel intimidating, but get the ball rolling and incrementally improve/expand your experience.

References: I use docker and Kubernetes daily. I currently manage numerous clusters and maintain pipelines for hundreds of microservices as I type this. I've been converting microservices into Docker images for companies hundreds if not thousands of times by now over the last bunches of years. I am also an avid and passionate open-source evangelist and Kubernetes/DevOps consultant. I author some Kubernetes controllers such as the Volume Autoscaler and have a set of Open Source Helm Charts and I love to contribute code/fixes wherever I run into issues.

Old justifications for this were to resize drives but all major cloud providers support handling the resizing operation for you now. You still need to trigger the resize. But with a controller like the Kubernetes Volume Autoscaler you don’t even need to do that!

An new open-source Kubernetes controller, the Kubernetes Volume Autoscaler, which auto-resizes your Persistent Volumes when they get almost full

Kubernetes Volume Autoscaler - An Kubernetes Controller to automatically scale up volumes (disks). I just recently released an update based on some feedback, adding Prometheus metrics support and fixing a few bugs

Autoscale everything. There’s no over provisioning if it just provisions as needed. HPA and Cluster Autoscaler and for disks I wrote and use the Volume Autoscaler. Nodes disappear as needed and appear as needed. I generally even do spot instances in production. All assuming you are using a cloud provider.

Volume Autoscaler - Automatically scale up your disks size, keeping your costs low and allowing you to grow over time. Also making one less thing your sysops/devops person has to do. (Shameless plug, I wrote this)