rust-prometheus VS cadvisor

I used the prometheus library because it was the most popular and easiest to find. Some unusual and unexpected aspects were that, to delete metrics, all labels must be specified at once, and to record metrics, I always needed the full list of labels that were used during creation. However, these issues were manageable.

[dependencies] prometheus = { git = "https://github.com/tikv/rust-prometheus", rev = "ac86a264223c8d918a43e739ca3c48bb4aaedb90", features = ["process"] }

Metrics exposition is implemented with prometheus crate.

The new Elasticsearch exporter is written in the Rust programming language and is open-sourced on GitHub: github.com/vinted/elasticsearch-exporter-rs. The exporter uses asynchronous Tokio runtime, Rust Prometheus instrumentation library and the official Elasticsearch client library. Metrics collection is decoupled from the serving /metrics endpoint. In addition, Elasticsearch time-based metrics in milliseconds are converted into seconds to comply with Prometheus best practices (metrics ending in “millis” are replaced by “seconds”, “_bytes” and “_seconds” and postfixes are added where appropriate).

Rust has a popular binding: https://github.com/tikv/rust-prometheus

cAdvisor

https://github.com/google/cadvisor exports Prometheus metrics, but also offers a simple web-ui for container metrics.

version: '3' volumes: prometheus-data: driver: local grafana-data: driver: local services: prometheus: image: prom/prometheus:latest container_name: prometheus ports: - "9090:9090" volumes: - /etc/prometheus:/etc/prometheus - prometheus-data:/prometheus restart: unless-stopped command: - "--config.file=/etc/prometheus/prometheus.yml" grafana: image: grafana/grafana:latest container_name: grafana ports: - "3000:3000" volumes: - grafana-data:/var/lib/grafana restart: unless-stopped node_exporter: image: quay.io/prometheus/node-exporter:latest container_name: node_exporter command: - '--path.rootfs=/host' pid: host restart: unless-stopped volumes: - '/:/host:ro,rslave' cadvisor: # TODO: latest tag is not updated, check latest release https://github.com/google/cadvisor/releases image: gcr.io/cadvisor/cadvisor-arm:v0.47.0 container_name: cadvisor ports: - "8080:8080" network_mode: host volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:ro privileged: true restart: unless-stopped depends_on: - redis redis: image: redis:latest container_name: redis ports: - "6379:6379"

For CPU and memory metrics, you can use cAdvisor to collect container level data.

Perhaps https://github.com/google/cadvisor + prometheus (or influx or whatever else) + grafana?

We could have used a much more focussed tool like Prometheus or Cadvisor to gather system stats, but that is not the main objective of this article.

Result: Interestingly, the GitHub Issue that talks about cAdvisor and Docker Desktop for Mac for running cAdvisor is still open and not fixed.

If you're running things under systemd, you can enable process accounting and use cAdvisor.

Check this thread if you’re having difficulties, but it might run out of the box these days: https://github.com/google/cadvisor/issues/1846