helm-charts VS kube-prometheus

The go to solution for a k8s monitoring setup is https://github.com/prometheus-community/helm-charts/tree/mai...

That's why Kube-Prometheus-Stack was created. It installs a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, providing an easy-to-operate, end-to-end Kubernetes cluster monitoring solution with Prometheus using the Prometheus Operator.

Step 1: Installing Kube-prometheus-stack

Kubernetes Documentation: https://kubernetes.io/docs/home/ Kubernetes Tutorials: https://kubernetes.io/docs/tutorials/ Kubernetes Community: https://kubernetes.io/community/ Prometheus: https://prometheus.io/ Grafana: https://grafana.com/ Elasticsearch: https://www.elastic.co/elasticsearch/ Kibana: https://www.elastic.co/kibana Helm: https://helm.sh/ Prometheus Helm Chart: https://github.com/prometheus-community/helm-charts/tree/main/prometheus Grafana Helm Chart: https://github.com/grafana/helm-charts/tree/main/grafana Elasticsearch Helm Chart: https://github.com/elastic/helm-charts/tree/main/elasticsearch Kibana Helm Chart: https://github.com/elastic/helm-charts/tree/main/kibana RBAC: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/ StatefulSets: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ DaemonSets: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ Taints and Tolerations: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ Custom Resource Definitions (CRDs): https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ Operators: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

Installation example for prometheus:

Prometheus can be deployed using the Prometheus Helm Chart. This helm chart contains a lot of features such as the already mentioned Push Gateway, Alert Manager and so on. For simplicity reasons of this tutorial I will not show all the Helm chart configuration but you can see a real example used by me here.

Building upon Bartłomiej Płotka's insightful blog on Prometheus and its passthrough agent mode, this post dives into implementing multi-cluster Prometheus support. Notably, the official inclusion of support in the widely-used kube-prometheus-stack came with the release in July 2023, making it easier to extend Prometheus monitoring across clusters.

The first thing you'll need, of course, is at least one backend to store metrics. So install Prometheus now:

The last one is mostly an observability stack with Prometheus, Metric server, and Prometheus adapter to have excellent insights into what is happening on the cluster. You can reuse the same stack for autoscaling by repurposing all the data collected for monitoring.

kustomize Kube Prometheus

On the other hand, the Kube-prometheus project provides documentation and scripts to operate end-to-end Kubernetes cluster monitoring using the Prometheus Operator, making easier the process of monitoring the Kubernetes cluster.

For our load testing we’ve deployed Temporal on Kubernetes, and we’re using MySQL for the persistence backend. The MySQL instance has 4 CPU cores and 32GB RAM, and each Temporal service (Frontend, History, Matching, and Worker) has 2 pods, with requests for 1 CPU core and 1GB RAM as a starting point. We’re not setting CPU limits for our pods—see our upcoming Temporal on Kubernetes post for more details on why. For monitoring we’ll use Prometheus and Grafana, installed via the kube-prometheus stack, giving us some useful Kubernetes metrics.

The 2 most common approaches I have seen are kube-prometheus-stack and kube-prometheus..

- I'm missing a lot of the Grafana dashboards that are provisioned during the deployment, not sure why as it has worked before, and wanted to add them after install... I believe it's different ConfigMaps like the one in kube-prometheus but I was wondering if there's a way to force provisioning them all again at once (multiple k8s, node_exporter, vm, etc)?

Check out the kube Prometheus project -- https://github.com/prometheus-operator/kube-prometheus It's a bit heavy, but the included recording rules and dashboards give you a great start at understanding your cluster.

The actual link to the prometheus/grafana bundle: https://github.com/prometheus-operator/kube-prometheus

Here’s a list of what’s installed: https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests