Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Follow Strimzi quickstart to install kafka in knative-eventing namespace, but use different Kafka cluster definition, see below. Knative workloads are expecting to be run in knative-eventing namespace, otherwise issues arise. And it's easier to keep Knative and Kafka in one namespace. Use kafka-cluster.yaml as kafka cluster resource instead of the one used in Strimzi quickstart (kafka-single-persistent.yaml). If you're not limited on disk, best to set storage: size: 50Gi or 100Gb in kafka-cluster yaml, and at least 25Gb for zookeeper storage. For trial quota, you're limited to 20Gb and 10Gb for zookeeper (if we're doing 2 Kafka clusters, if one - can be more).
Install scaling controller for Kafka sources - Keda autoscaler. HPA parameters are controlled by annotations on the Kafka source yaml definition:
Knative exposes a couple of it's own metrics (like processing delays) and also exposes a huge amount of Kafka metrics for it's consumers/producers. I ended up curl-ing Knative Services on the metrics port, and scripting a tool that would help to create primitive Grafana dashboard for the list of metric names and uid of datasource. See readme on how to use the tool. Or can replace datasource uid in the dashboard-*.json with your datasource uid, and make sure job selectors in the dashboard JSON match the service name that sends metrics.
Knative dashboards together with Kafka's dashboards it sheds light on almost any aspect of what's going on in the system.