node_exporter VS cadvisor

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz $ tar -xzvf node_exporter-1.7.0.linux-amd64.tar.gz

Node Exporter

The Prometheus node_exporter can provide this information, doesn't require root. You could run it as a systemd user unit if you don't have root.

I personally find it best to learn through experimentation. Start with reading a bit about Prometheus and Grafana through their docs, and then familiarise yourself with setting up a local Prometheus + Grafana instance either locally or with docker using docker-compose, along with something to generate /metrics endpoint for Prometheus to scrape from such as a custom Prometheus exporter in Python or using Node Exporter.

Your best bet might be to fork node_exporter to get you more verbose socket stats: https://github.com/prometheus/node_exporter/blob/master/collector/sockstat_linux.go

I use Grafana + Prometheus + Node Exporter.

Crazy idea, can't NX devices run Docker? I wonder if the node_exporter would work.

Kubernetes itself will not notify you, the way I've seen people do this, is to use something like kube-state-metrics or node_exporter, export that to Prometheus (or preferrably VictoriaMetrics because Prometheus is terrible IMO), and then setup alarms on that with alertmanager or equivalent, or just look at dashboards regularly with Grafana. Realistically I recommend only setting alerts on disk usage and application/database latency. CPU and memory utilization isn't a great metric to alert on a lot of the time.

You may setup node exporter and collect metrics with prometheus for example. Its not quite "simple" way, but still you may find it useful.

cAdvisor

https://github.com/google/cadvisor exports Prometheus metrics, but also offers a simple web-ui for container metrics.

version: '3' volumes: prometheus-data: driver: local grafana-data: driver: local services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - "9090:9090" volumes: - /etc/prometheus:/etc/prometheus - prometheus-data:/prometheus restart: unless-stopped command: - "--config.file=/etc/prometheus/prometheus.yml" grafana: image: grafana/grafana:latest container_name: grafana ports: - "3000:3000" volumes: - grafana-data:/var/lib/grafana restart: unless-stopped node_exporter: image: quay.io/prometheus/node-exporter:latest container_name: node_exporter command: - '--path.rootfs=/host' pid: host restart: unless-stopped volumes: - '/:/host:ro,rslave' cadvisor: # TODO: latest tag is not updated, check latest release https://github.com/google/cadvisor/releases image: gcr.io/cadvisor/cadvisor-arm:v0.47.0 container_name: cadvisor ports: - "8080:8080" network_mode: host volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:ro privileged: true restart: unless-stopped depends_on: - redis redis: image: redis:latest container_name: redis ports: - "6379:6379"

For CPU and memory metrics, you can use cAdvisor to collect container level data.

Perhaps https://github.com/google/cadvisor + prometheus (or influx or whatever else) + grafana?

We could have used a much more focussed tool like Prometheus or Cadvisor to gather system stats, but that is not the main objective of this article.

Result: Interestingly, the GitHub Issue that talks about cAdvisor and Docker Desktop for Mac for running cAdvisor is still open and not fixed.

If you're running things under systemd, you can enable process accounting and use cAdvisor.

Check this thread if you’re having difficulties, but it might run out of the box these days: https://github.com/google/cadvisor/issues/1846