kube-prometheus VS kube-state-metrics

The last one is mostly an observability stack with Prometheus, Metric server, and Prometheus adapter to have excellent insights into what is happening on the cluster. You can reuse the same stack for autoscaling by repurposing all the data collected for monitoring.

kustomize Kube Prometheus

On the other hand, the Kube-prometheus project provides documentation and scripts to operate end-to-end Kubernetes cluster monitoring using the Prometheus Operator, making easier the process of monitoring the Kubernetes cluster.

For our load testing we’ve deployed Temporal on Kubernetes, and we’re using MySQL for the persistence backend. The MySQL instance has 4 CPU cores and 32GB RAM, and each Temporal service (Frontend, History, Matching, and Worker) has 2 pods, with requests for 1 CPU core and 1GB RAM as a starting point. We’re not setting CPU limits for our pods—see our upcoming Temporal on Kubernetes post for more details on why. For monitoring we’ll use Prometheus and Grafana, installed via the kube-prometheus stack, giving us some useful Kubernetes metrics.

The 2 most common approaches I have seen are kube-prometheus-stack and kube-prometheus..

- I'm missing a lot of the Grafana dashboards that are provisioned during the deployment, not sure why as it has worked before, and wanted to add them after install... I believe it's different ConfigMaps like the one in kube-prometheus but I was wondering if there's a way to force provisioning them all again at once (multiple k8s, node_exporter, vm, etc)?

Check out the kube Prometheus project -- https://github.com/prometheus-operator/kube-prometheus It's a bit heavy, but the included recording rules and dashboards give you a great start at understanding your cluster.

The actual link to the prometheus/grafana bundle: https://github.com/prometheus-operator/kube-prometheus

Here’s a list of what’s installed: https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests

Kube-State-Metrics is an agent service that listens to the Kubernetes API and generates metrics about the state of cluster objects, such as deployments, nodes, and pods. Unlike resource metrics (like CPU or memory usage), it provides insights into object-level data, such as the number of replicas or pod status, making it useful for monitoring the desired versus actual state of Kubernetes components.

With Kube-State metrics, you get a complete view of the core components and overall health of your Kubernetes clusters. You can monitor the real-time state and quickly identify any issues or bottlenecks. Detailed container-level metrics allow you to visually navigate through different layers of your cluster, making it easier to spot problems like memory leaks in individual containers. This helps you resolve issues faster and more efficiently.

Look for example at the metrics exposed by kube state metrics: https://github.com/kubernetes/kube-state-metrics/tree/main/d...

With controllers metrics + kube state metrics about most Kubernetes resources, you can easily build alerts when a resource fails to reconcile.

> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"

Take a look at KEDA, it's exactly this: https://keda.sh/

git clone https://github.com/kubernetes/kube-state-metrics.git

https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md According to this github repo completion is responsible of termination date if I correctly understood .

kube-state-metrics

Those are entirely different types of data. You can get that from something like kube-state-metrics

I've never had a cluster of that size, so take it with a grain of salt - but maybe you could try purpose-based sharding? KSM has allowlist and denylist config flags, for configuring which metrics it exposes https://github.com/kubernetes/kube-state-metrics/blob/main/docs/cli-arguments.md