postgres-operator VS kube-state-metrics

As for the Operators, I've been using Crunchy PGO, which is very high quality, and one of the most widely used. You can install it via Helm, or via OLM from OperatorHub. There are other good ones as well, but none that I have experience with. The only issue I've run into so far is I've had to disable TLS on the database cluster, as Prowlarr refused to connect with it for some reason (Radarr was fine). I still need to open an issue with the Prowlarr team about that, but I might switch to a service mesh for TLS anyway.

The Crunchy operator, seemingly like most (if not all) of the other Postgres operators (Zalando, KubeDB, and StackGres, etc.), is essentially a wrapper for Patroni. IMO if someone wanted a Patroni cluster, they would just build one. The point of an operator is to manage the cluster resources and node relationships, so why not have it take the role Patroni is filling here? It's already reaching into the nodes, obtaining status, managing the routing, etc., so why add the extra layer?

On the topic of Postgres, you should look into an operator or Helm chart that can setup common things (like replication and auto-failover), such as Crunchy's Postgres operator, or consider using a "cloud-native" distributed database like CockroachDB (disclaimer: I am a Cockroach Labs employee) which has its own operator as well. Another word of warning, running stateful services, particularly mission critical databases, can require a lot of maintenance work (it's my full-time job), so unless this is for a hobby project, I would highly recommend you look into using a managed database offerring. Every major cloud provider and most database companies have one.

For postgres you can also have a look at PGO or bitnami helm chart

Let's say a new Kubernetes version comes out in April. In November, as everything works perfectly well, you decide to install a Postgres operator on it. Bummer, it doesn't work. It's not a huge issue, you just wait until the bug is resolved (already done[0]), but it's just one of these tiny things that I don't get when running Postrges natively. And I'm saying this as a big fan of Crunchy Data running some production loads on it without a failure for quite some time now.

[0] https://github.com/CrunchyData/postgres-operator/issues/3476

There is one particular client that have a somewhat big database 40-120gb (it change size over the year), and for that we used CrunchyData Postgres operator ( https://access.crunchydata.com/documentation/postgres-operator/v5/ ) we have no commercial relation with them, but oboi let me tell you the god send that thing is, this database in specific process massive data and it is distributed between several nodes in a read-write and read-only set, and let me tell you, it is amazing how easy it is to move things around, take backups, increase the capacity and a bunch of other goodies that operator bring. Give it a try.

git clone https://github.com/kubernetes/kube-state-metrics.git

https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md According to this github repo completion is responsible of termination date if I correctly understood .

kube-state-metrics

Those are entirely different types of data. You can get that from something like kube-state-metrics

I've never had a cluster of that size, so take it with a grain of salt - but maybe you could try purpose-based sharding? KSM has allowlist and denylist config flags, for configuring which metrics it exposes https://github.com/kubernetes/kube-state-metrics/blob/main/docs/cli-arguments.md

Kubernetes itself will not notify you, the way I've seen people do this, is to use something like kube-state-metrics or node_exporter, export that to Prometheus (or preferrably VictoriaMetrics because Prometheus is terrible IMO), and then setup alarms on that with alertmanager or equivalent, or just look at dashboards regularly with Grafana. Realistically I recommend only setting alerts on disk usage and application/database latency. CPU and memory utilization isn't a great metric to alert on a lot of the time.

Another tip that you could consider spelling out a little more, is to monitor the number of resources created by Kind. This is somewhat mentioned for jobs and services, but any Kind of which thousands of resources are created will put stress on the control-plane. The total number of resources per namespace/cluster can be monitored with kube-state-metrics. KSM does not emit metrics of resources created from CRDs. These metrics can be implemented with KSM's custom resource state metrics: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md

We then have various other Metrics called Kubernetes Workload Metrics. These are the dashboards with names that start with “Kubernetes / Compute Resources / Workload”. These dashboards are specific to the services you are running. They take into account the Kubernetes Workloads in your various namespaces, using kube-state-metrics. For a closer look, check out otel_demo_app_k8s_dashboard.tf.

The first step to optimizing costs is gaining visibility into your costs using tools. Kubernetes provides a Metrics Server and kube-state-metrics that can give you the overall picture of resource utilization by your cluster. There are more tools that provide more granular breakdowns and provide dashboards with business metrics, infra cost, and alerting functionalities. Here are some strategies to optimize your resource utilization and cloud bills on k8s.