zeebe VS keda

You can Ctrl + F (or Command + F) your way through the file, but you have to know what you are looking for.

Zeebe

For this purpose, it's the equivalent of using a sledge hammer to crack a walnut, but I think something like Zeebe could facilitate that

I’m doing a lot of exploratory work with BPMN right now.

I think if your business is largely transactional (think Stripe), there is a lot of value to be had by framing your development as “business process automation”.

The term (and BPMN) has a lot of enterprise baggage, but some of the tools out there [0][1] are well suited to orchestrating services (and people where necessary) as a single automated process. The the ability to build that flow visually using BPMN, and then execute it in a workflow engine where you can monitor it, audit it, and optimize over time is pretty compelling.

Here’s an interesting read on the topic: https://www.infoq.com/articles/events-workflow-automation/

[0] https://github.com/camunda-cloud/zeebe

[1] https://github.com/flowable/flowable-engine

(I have no idea whether ES is used in temporal though… never looked at the code)

[1] https://github.com/camunda-cloud/zeebe

[2] https://temporal.io/

It seems you want something like KEDA (https://keda.sh)

Most just utilize out of the box macro resources available in HPA.

For more advanced use cases there is keda - https://keda.sh/

Thankfully KEDA operator was already part of the cluster, and all Robin had to do was create a ScaledObject manifest targeting the Dispatch ScaleUp event, based on the rabbitmq_global_messages_received_total metric from Prometheus.

Keda

KEDA introduces event-driven scaling to Kubernetes workloads. It integrates with Kubernetes Horizontal Pod Autoscalers and can scale pods based on external metrics from services like databases and message queues (Kafka, RabbitMQ, MongoDB).

# update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml

This is traditionally a AWS Lambda usecase - or an OpenFaas kind of usecase. But very recently i discovered https://keda.sh/ and it seems it is specifically meant for this in a kubernetes environment.

If you deploy the application to Amazon EKS, you can also consider using KEDA to auto-scale your consumer application based on the number of messages in the MSK topic.