systemd_exporter VS cadvisor

Worth looking into prometheus. in basic form that'd be gathering metrics from https://github.com/prometheus-community/systemd_exporter on your hosts, and configure alerting in grafana or prometheus alertmanager to notify when a threshold is exceeded

Before you get into process monitoring, you probably want to look at cgroup monitoring. Since most services are started by systemd units, you can use the systemd_exporter or cAdvisor to track process groups.

If you want to monitor services, you might want to look at leveraging cgroup-based data with cAdvisor or the systemd_exporter.

systemd_exporer

cAdvisor

https://github.com/google/cadvisor exports Prometheus metrics, but also offers a simple web-ui for container metrics.

version: '3' volumes: prometheus-data: driver: local grafana-data: driver: local services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - "9090:9090" volumes: - /etc/prometheus:/etc/prometheus - prometheus-data:/prometheus restart: unless-stopped command: - "--config.file=/etc/prometheus/prometheus.yml" grafana: image: grafana/grafana:latest container_name: grafana ports: - "3000:3000" volumes: - grafana-data:/var/lib/grafana restart: unless-stopped node_exporter: image: quay.io/prometheus/node-exporter:latest container_name: node_exporter command: - '--path.rootfs=/host' pid: host restart: unless-stopped volumes: - '/:/host:ro,rslave' cadvisor: # TODO: latest tag is not updated, check latest release https://github.com/google/cadvisor/releases image: gcr.io/cadvisor/cadvisor-arm:v0.47.0 container_name: cadvisor ports: - "8080:8080" network_mode: host volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:ro privileged: true restart: unless-stopped depends_on: - redis redis: image: redis:latest container_name: redis ports: - "6379:6379"

For CPU and memory metrics, you can use cAdvisor to collect container level data.

Perhaps https://github.com/google/cadvisor + prometheus (or influx or whatever else) + grafana?

We could have used a much more focussed tool like Prometheus or Cadvisor to gather system stats, but that is not the main objective of this article.

Result: Interestingly, the GitHub Issue that talks about cAdvisor and Docker Desktop for Mac for running cAdvisor is still open and not fixed.

If you're running things under systemd, you can enable process accounting and use cAdvisor.

Check this thread if you’re having difficulties, but it might run out of the box these days: https://github.com/google/cadvisor/issues/1846