mop VS keda

While Pulsar has its own protocol that handles streaming, queues, and a lot of the other features (distributed transactions, functions, etc.), it can also speak other messaging protocols via plug-ins. Looking around, the ones that appear to be actively developed are MQTT, Kafka (so your existing applications that use Kafka can also use Pulsar), AMQP, and JMS.

If you need a highly available broker you can consider https://github.com/streamnative/mop which adds a MQTT protocol handler to Apache Pulsar.

I recently contributed auth support and TLS support also just landed.

It seems you want something like KEDA (https://keda.sh)

Most just utilize out of the box macro resources available in HPA.

For more advanced use cases there is keda - https://keda.sh/

Thankfully KEDA operator was already part of the cluster, and all Robin had to do was create a ScaledObject manifest targeting the Dispatch ScaleUp event, based on the rabbitmq_global_messages_received_total metric from Prometheus.

Keda

KEDA introduces event-driven scaling to Kubernetes workloads. It integrates with Kubernetes Horizontal Pod Autoscalers and can scale pods based on external metrics from services like databases and message queues (Kafka, RabbitMQ, MongoDB).

# update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml

This is traditionally a AWS Lambda usecase - or an OpenFaas kind of usecase. But very recently i discovered https://keda.sh/ and it seems it is specifically meant for this in a kubernetes environment.

If you deploy the application to Amazon EKS, you can also consider using KEDA to auto-scale your consumer application based on the number of messages in the MSK topic.