charts VS kube-state-metrics

I'm using the official stable/prometheus-operator chart do deploy Prometheus with helm.

Enter the default username: admin and password: prom-operator which you can find from here to access Grafana.

In this step opta generates terraform code to deploys helm chart. You can review the plan and accept it. It will take around 2-3 minutes for the plan to get applied. For specifying values to the helm chart you can modify the values inside opta-gcp/opta.yml . Refer to jenkins helm chart for more details https://github.com/helm/charts/tree/master/stable/jenkins. After opta apply finishes you will have jenkins up and running on your infra.

I do use Helm to deploy, but the trouble with using it for secrets is that it overwrites them on an upgrade.

In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

Redis: I have used Azure Cache for Redis, but feel free to explore other options e.g. you can install one in your Kubernetes cluster using a Helm chart).

To install Prometheus, the most convenient way is using the Prometheus Operator Helm Chart. Helm is a package manager for Kubernetes. It simplifies the installation by integrating most of the necessary configuration settings into one package. The following instructions are based on a blog post which describes the installation process specific for AKS clusters.

Kubernetes itself will not notify you, the way I've seen people do this, is to use something like kube-state-metrics or node_exporter, export that to Prometheus (or preferrably VictoriaMetrics because Prometheus is terrible IMO), and then setup alarms on that with alertmanager or equivalent, or just look at dashboards regularly with Grafana. Realistically I recommend only setting alerts on disk usage and application/database latency. CPU and memory utilization isn't a great metric to alert on a lot of the time.

We then have various other Metrics called Kubernetes Workload Metrics. These are the dashboards with names that start with “Kubernetes / Compute Resources / Workload”. These dashboards are specific to the services you are running. They take into account the Kubernetes Workloads in your various namespaces, using kube-state-metrics. For a closer look, check out otel_demo_app_k8s_dashboard.tf.

The first step to optimizing costs is gaining visibility into your costs using tools. Kubernetes provides a Metrics Server and kube-state-metrics that can give you the overall picture of resource utilization by your cluster. There are more tools that provide more granular breakdowns and provide dashboards with business metrics, infra cost, and alerting functionalities. Here are some strategies to optimize your resource utilization and cloud bills on k8s.

If you have the kube-state-metrics + Prometheus with a sufficiently long retention time, then you could write a PromQL query based on the "kube_pod_created" metric to get the answer to your question. It might be possible with the metrics-server + Prometheus, but I think it would not count some pods that live less than one scrape interval.

There is, also developed by k8s team, the kube-state-metrics, a service that listens to the Kubernetes API server and generates metrics about the state of the objects such as deployments, nodes, and pods.

2. kube-state-metrics

I am using kube-state-metrics with Prometheus and we have these metrics available: https://github.com/kubernetes/kube-state-metrics/blob/master/docs/cronjob-metrics.md