helm-charts VS kubernetes-mixin

Prometheus can be deployed using the Prometheus Helm Chart. This helm chart contains a lot of features such as the already mentioned Push Gateway, Alert Manager and so on. For simplicity reasons of this tutorial I will not show all the Helm chart configuration but you can see a real example used by me here.

Building upon Bartłomiej Płotka's insightful blog on Prometheus and its passthrough agent mode, this post dives into implementing multi-cluster Prometheus support. Notably, the official inclusion of support in the widely-used kube-prometheus-stack came with the release in July 2023, making it easier to extend Prometheus monitoring across clusters.

The first thing you'll need, of course, is at least one backend to store metrics. So install Prometheus now:

For the request following, something like jeager https://www.jaegertracing.io/, because you are talking more about tracing than necessarily logging. For just monitoring, https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack would be the starting point, then it depends. Nginx gives metrics out of the box, then you can pull in the dashboard like https://grafana.com/grafana/dashboards/14314-kubernetes-nginx-ingress-controller-nextgen-devops-nirvana/ , or full metal with something like service mesh monitoring which would provably fulfil most of the requirements

kube-prometheus-stack is a Helm chart that contains several components to monitor the Kubernetes cluster, along with Grafana dashboards Grafana Dashboards to visualize the data. This option will be used in this article.

This one I highly recommend: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

We use kube-prometheus-stack for metrics and have added the K8s views dashboards from grafana-dashboards-kubernetes. You should check out the k8s-views-global dashboard. I believe it's just what you are looking for.

You should take a look at "kube-prometheus-stack". It not only includes prometheus, node-exporter and Grafana but also a ton of preconfigured alerts and dashboards. Will save you a lot of work!

Since their are different ways to install what exactly did you install? Vanilla charts , stack, operator? https://github.com/prometheus-community/helm-charts/tree/main/charts

I'd suggest looking at the queries used by the kubernetes mixin dashboards for inspiration.

Came across this lib, but realized the last update was 4 years ago.

Take a look at the kubernetes mixins to get an idea and/or starting point

The last step is to use the Mixin dashboard to visualize the usage of PV, you can get it from here

It's working good so far, except for the annoying CPUThrottlingHigh alert that is firing for many pods (including the own Prometheus' config-reloaders containers). This alert is currently under discussion, and I want to silence its notifications for now.

I think it might still have issues figuring out that it needs to apply CRDs first: https://github.com/grafana/tanka/issues/246 Besides that, I found it super-handy for deploying https://github.com/prometheus-operator/prometheus-operator and https://github.com/kubernetes-monitoring/kubernetes-mixin

These all standard dashboards are basically generated from the kubernetes-mixin project.

I would suggest looking at the Kubernetes Mixin: https://github.com/kubernetes-monitoring/kubernetes-mixin, it has both alerts and Grafana dashboards.