kube-prometheus VS local-path-provisioner

The last one is mostly an observability stack with Prometheus, Metric server, and Prometheus adapter to have excellent insights into what is happening on the cluster. You can reuse the same stack for autoscaling by repurposing all the data collected for monitoring.

kustomize Kube Prometheus

On the other hand, the Kube-prometheus project provides documentation and scripts to operate end-to-end Kubernetes cluster monitoring using the Prometheus Operator, making easier the process of monitoring the Kubernetes cluster.

For our load testing we’ve deployed Temporal on Kubernetes, and we’re using MySQL for the persistence backend. The MySQL instance has 4 CPU cores and 32GB RAM, and each Temporal service (Frontend, History, Matching, and Worker) has 2 pods, with requests for 1 CPU core and 1GB RAM as a starting point. We’re not setting CPU limits for our pods—see our upcoming Temporal on Kubernetes post for more details on why. For monitoring we’ll use Prometheus and Grafana, installed via the kube-prometheus stack, giving us some useful Kubernetes metrics.

The 2 most common approaches I have seen are kube-prometheus-stack and kube-prometheus..

- I'm missing a lot of the Grafana dashboards that are provisioned during the deployment, not sure why as it has worked before, and wanted to add them after install... I believe it's different ConfigMaps like the one in kube-prometheus but I was wondering if there's a way to force provisioning them all again at once (multiple k8s, node_exporter, vm, etc)?

Check out the kube Prometheus project -- https://github.com/prometheus-operator/kube-prometheus It's a bit heavy, but the included recording rules and dashboards give you a great start at understanding your cluster.

The actual link to the prometheus/grafana bundle: https://github.com/prometheus-operator/kube-prometheus

Here’s a list of what’s installed: https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests

Deploy local-path-provisioner storage class but it does not support readwritemany so for high availability of your Kubernetes cluster better to use longhorn

I use this one. I'm waiting for the day it's combined with syncthing to sync across all nodes. https://github.com/rancher/local-path-provisioner

So I have this single node k3s cluster. k3s uses local-path (https://github.com/rancher/local-path-provisioner) as default SC that allows one to create dynamic volumes using nodes local storage.

Just create 1 single partition and format it with whatevery filesystem you like. And then use ranchers local-path-provisioner which will create a folder per PV (k3s has this integrated by default).

Tinkering locally with local path provisioner (https://github.com/rancher/local-path-provisioner), I find that I can delete and re-create the pod, and the data persists on disk. However, if I delete the PVC, when I recreate the PVC, a new directory on disk is created.

It is better to use https://github.com/rancher/local-path-provisioner (or similar) for this case which will do PVC on local directories because manually linking PV<>PVC will not work.

Since you only have one physical node anyway, I would just make the cluster a single-node cluster (1 VM) and use local storage on that VM. I’m biased though because this is what I do (I run K3s and use local path provisioner).

Rancher's Local Path Provisioner - From reading, seems to just use HostPath or Local PVs under the hood, but adds dynamic provisoning

With any of those tools, you'd implement a network storage on top of a network storage. I would go with mouting few volumes per node +local storage like (https://github.com/rancher/local-path-provisioner).