k3d-nfs-dynamic-volumes VS keda

Now, for the purposes of this article, in case you don't have an NFS server available, we will use a simple NFS Server Provisioner, which we'll use only for example purposes. As mentioned before, using a managed solution from a cloud provider or a properly configured HA NFS server in your infrastructure is highly recommended. We'll install not the most up-to-date solution, but it should work for example purposes. We will follow the Quickstart found in the repo, mixed with this repo which does some small tweaks to make it work with K3d, which is summarized in the following commands run from the helm folder:

It seems you want something like KEDA (https://keda.sh)

Most just utilize out of the box macro resources available in HPA.

For more advanced use cases there is keda - https://keda.sh/

Thankfully KEDA operator was already part of the cluster, and all Robin had to do was create a ScaledObject manifest targeting the Dispatch ScaleUp event, based on the rabbitmq_global_messages_received_total metric from Prometheus.

Keda

KEDA introduces event-driven scaling to Kubernetes workloads. It integrates with Kubernetes Horizontal Pod Autoscalers and can scale pods based on external metrics from services like databases and message queues (Kafka, RabbitMQ, MongoDB).

# update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml

This is traditionally a AWS Lambda usecase - or an OpenFaas kind of usecase. But very recently i discovered https://keda.sh/ and it seems it is specifically meant for this in a kubernetes environment.

If you deploy the application to Amazon EKS, you can also consider using KEDA to auto-scale your consumer application based on the number of messages in the MSK topic.