cadvisor VS kube-state-metrics

cAdvisor is a container monitoring tool that provides detailed metrics about Docker containers, including network usage.

Find more details here

cAdvisor

https://github.com/google/cadvisor exports Prometheus metrics, but also offers a simple web-ui for container metrics.

version: '3' volumes: prometheus-data: driver: local grafana-data: driver: local services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - "9090:9090" volumes: - /etc/prometheus:/etc/prometheus - prometheus-data:/prometheus restart: unless-stopped command: - "--config.file=/etc/prometheus/prometheus.yml" grafana: image: grafana/grafana:latest container_name: grafana ports: - "3000:3000" volumes: - grafana-data:/var/lib/grafana restart: unless-stopped node_exporter: image: quay.io/prometheus/node-exporter:latest container_name: node_exporter command: - '--path.rootfs=/host' pid: host restart: unless-stopped volumes: - '/:/host:ro,rslave' cadvisor: # TODO: latest tag is not updated, check latest release https://github.com/google/cadvisor/releases image: gcr.io/cadvisor/cadvisor-arm:v0.47.0 container_name: cadvisor ports: - "8080:8080" network_mode: host volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:ro privileged: true restart: unless-stopped depends_on: - redis redis: image: redis:latest container_name: redis ports: - "6379:6379"

For CPU and memory metrics, you can use cAdvisor to collect container level data.

Perhaps https://github.com/google/cadvisor + prometheus (or influx or whatever else) + grafana?

We could have used a much more focussed tool like Prometheus or Cadvisor to gather system stats, but that is not the main objective of this article.

With Kube-State metrics, you get a complete view of the core components and overall health of your Kubernetes clusters. You can monitor the real-time state and quickly identify any issues or bottlenecks. Detailed container-level metrics allow you to visually navigate through different layers of your cluster, making it easier to spot problems like memory leaks in individual containers. This helps you resolve issues faster and more efficiently.

Look for example at the metrics exposed by kube state metrics: https://github.com/kubernetes/kube-state-metrics/tree/main/d...

With controllers metrics + kube state metrics about most Kubernetes resources, you can easily build alerts when a resource fails to reconcile.

> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"

Take a look at KEDA, it's exactly this: https://keda.sh/

git clone https://github.com/kubernetes/kube-state-metrics.git

https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md According to this github repo completion is responsible of termination date if I correctly understood .

kube-state-metrics

Those are entirely different types of data. You can get that from something like kube-state-metrics

I've never had a cluster of that size, so take it with a grain of salt - but maybe you could try purpose-based sharding? KSM has allowlist and denylist config flags, for configuring which metrics it exposes https://github.com/kubernetes/kube-state-metrics/blob/main/docs/cli-arguments.md

Kubernetes itself will not notify you, the way I've seen people do this, is to use something like kube-state-metrics or node_exporter, export that to Prometheus (or preferrably VictoriaMetrics because Prometheus is terrible IMO), and then setup alarms on that with alertmanager or equivalent, or just look at dashboards regularly with Grafana. Realistically I recommend only setting alerts on disk usage and application/database latency. CPU and memory utilization isn't a great metric to alert on a lot of the time.