cluster-api VS k8s-prometheus-adapter

In this blog post, we will demonstrate how easy and fast it is to deploy Sveltos on an RKE2 cluster with the help of ArgoCD, register two RKE2 Cluster API (CAPI) clusters and create a ClusterProfile to deploy Prometheus and Grafana Helm charts down the managed CAPI clusters.

4. Having moved to a container orchestrator, all of my nodes are immutable. Hardware and VM instances _can_ be born magically into existence. Nearly all infra providers support [cluster-api](https://cluster-api.sigs.k8s.io/). Network infrastructure can now be managed with TF, so I go that route.

## Linux curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.4.4/clusterctl-linux-amd64 -o clusterctl sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl ## Mac brew install clusterctl

Did you ever try CAPI? https://github.com/kubernetes-sigs/cluster-api

You might find interesting the capi-rancher-import k8s operator we use in Sylva, it would adopt in Rancher server the Cluster API created k8s clusters (with bootstrap provider kubeadm or even rke2 - you can lookup CAPBR for the latter). I understand your clusters are not created by Cluster API, so if you could move the workloads/resources to new clusters created by Cluster API, this can come handy. (Adoption of non-CAPI clusters into CAPI is not yet a standard practice, more in https://github.com/kubernetes-sigs/cluster-api/issues/7776)

Most of the comments have mentioned older tools like kubespray, Ansible, Rancher etc. I would suggest the cloud native way using ClusterAPI or use a tool that relies on ClusterAPI in the backend called Talos

Cluster API

Sveltos is a powerful open source project that makes managing Kubernetes add-ons a breeze. It automatically discovers ClusterAPI powered clusters and allows you to easily register any other cluster (like GKE). Then, it seamlessly manages Kubernetes add-ons across all your clusters.

The last one is mostly an observability stack with Prometheus, Metric server, and Prometheus adapter to have excellent insights into what is happening on the cluster. You can reuse the same stack for autoscaling by repurposing all the data collected for monitoring.

I'm deploying kube-prometheus-stack along with prometheus-adapter in my monitoring stack for custom metrics.

I would like to see if anyone deployed prometheus-adapter and kube-prometheus-stack together for monitoring?

For us it is saturation of CPU and thread pool. It's implemented by exposing metrics of the thread pool to prometheus and turning that into a custom metric. (see) Looking at scaling based on job queue length next.

If you are using Prometheus or kube-prometheus-stack, you will need https://github.com/kubernetes-sigs/prometheus-adapter We are using it to scale our Pods based on number of messages in RabbitMQ queue. There also a walkthrough on https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/walkthrough.md

A prometheus-adapter installation.

Prometheus adapter to get custom/external metrics from Prometheus instance into Kubernetes API.

I never tested this but you have customized metrics API if the value % is available should work from my point of view Check this here https://github.com/kubernetes-sigs/prometheus-adapter

CPU and memory might not be the right metrics for your application to make scaling decisions. In such cases, you can use HPA (or VPA) with custom metrics as an alternative. To use custom metrics for autoscaling, you can use a custom metrics adapter instead of the Kubernetes Metrics Server. Popular custom metrics adapters are the Prometheus adapter and Kubernetes Event-Driven Autoscaler (KEDA).