strimzi-kafka-operator VS grafana-dashboard-from-metric-list

Knative dashboards together with Kafka's dashboards it sheds light on almost any aspect of what's going on in the system.

Strimzi is an operator for deploying Kafka in K8s, and makes extensive use of CRs. E.g., https://github.com/strimzi/strimzi-kafka-operator/blob/0.31.1/examples/kafka/kafka-persistent.yaml

But I encountered a problem. In the example files they provide prometheus-additional.yaml file which is referred to in prometheus.yaml as additionalScrapeConfigs field. It collects Kubernetes node metrics for which it has panels in example Grafana dashboards. All good. But the problem is, the prometheus-additional.yaml file has a field called namespaces: names: [] (File can be seen here - https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-additional-properties/prometheus-additional.yaml) in kubernetes_sd_configs.

Knative exposes a couple of it's own metrics (like processing delays) and also exposes a huge amount of Kafka metrics for it's consumers/producers. I ended up curl-ing Knative Services on the metrics port, and scripting a tool that would help to create primitive Grafana dashboard for the list of metric names and uid of datasource. See readme on how to use the tool. Or can replace datasource uid in the dashboard-*.json with your datasource uid, and make sure job selectors in the dashboard JSON match the service name that sends metrics.