keda VS k8s-prometheus-adapter

It seems you want something like KEDA (https://keda.sh)

Most just utilize out of the box macro resources available in HPA.

For more advanced use cases there is keda - https://keda.sh/

Thankfully KEDA operator was already part of the cluster, and all Robin had to do was create a ScaledObject manifest targeting the Dispatch ScaleUp event, based on the rabbitmq_global_messages_received_total metric from Prometheus.

Keda

KEDA introduces event-driven scaling to Kubernetes workloads. It integrates with Kubernetes Horizontal Pod Autoscalers and can scale pods based on external metrics from services like databases and message queues (Kafka, RabbitMQ, MongoDB).

# update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml

This is traditionally a AWS Lambda usecase - or an OpenFaas kind of usecase. But very recently i discovered https://keda.sh/ and it seems it is specifically meant for this in a kubernetes environment.

If you deploy the application to Amazon EKS, you can also consider using KEDA to auto-scale your consumer application based on the number of messages in the MSK topic.

The last one is mostly an observability stack with Prometheus, Metric server, and Prometheus adapter to have excellent insights into what is happening on the cluster. You can reuse the same stack for autoscaling by repurposing all the data collected for monitoring.

I'm deploying kube-prometheus-stack along with prometheus-adapter in my monitoring stack for custom metrics.

I would like to see if anyone deployed prometheus-adapter and kube-prometheus-stack together for monitoring?

For us it is saturation of CPU and thread pool. It's implemented by exposing metrics of the thread pool to prometheus and turning that into a custom metric. (see) Looking at scaling based on job queue length next.

If you are using Prometheus or kube-prometheus-stack, you will need https://github.com/kubernetes-sigs/prometheus-adapter We are using it to scale our Pods based on number of messages in RabbitMQ queue. There also a walkthrough on https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/walkthrough.md

A prometheus-adapter installation.

Prometheus adapter to get custom/external metrics from Prometheus instance into Kubernetes API.

I never tested this but you have customized metrics API if the value % is available should work from my point of view Check this here https://github.com/kubernetes-sigs/prometheus-adapter

CPU and memory might not be the right metrics for your application to make scaling decisions. In such cases, you can use HPA (or VPA) with custom metrics as an alternative. To use custom metrics for autoscaling, you can use a custom metrics adapter instead of the Kubernetes Metrics Server. Popular custom metrics adapters are the Prometheus adapter and Kubernetes Event-Driven Autoscaler (KEDA).