node_exporter VS alertmanager

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz $ tar -xzvf node_exporter-1.7.0.linux-amd64.tar.gz

Node Exporter

The Prometheus node_exporter can provide this information, doesn't require root. You could run it as a systemd user unit if you don't have root.

I personally find it best to learn through experimentation. Start with reading a bit about Prometheus and Grafana through their docs, and then familiarise yourself with setting up a local Prometheus + Grafana instance either locally or with docker using docker-compose, along with something to generate /metrics endpoint for Prometheus to scrape from such as a custom Prometheus exporter in Python or using Node Exporter.

Your best bet might be to fork node_exporter to get you more verbose socket stats: https://github.com/prometheus/node_exporter/blob/master/collector/sockstat_linux.go

I use Grafana + Prometheus + Node Exporter.

Crazy idea, can't NX devices run Docker? I wonder if the node_exporter would work.

Kubernetes itself will not notify you, the way I've seen people do this, is to use something like kube-state-metrics or node_exporter, export that to Prometheus (or preferrably VictoriaMetrics because Prometheus is terrible IMO), and then setup alarms on that with alertmanager or equivalent, or just look at dashboards regularly with Grafana. Realistically I recommend only setting alerts on disk usage and application/database latency. CPU and memory utilization isn't a great metric to alert on a lot of the time.

You may setup node exporter and collect metrics with prometheus for example. Its not quite "simple" way, but still you may find it useful.

- Alert Manager

You could use prometheus as a monitoring tool, blackbox_exporter to "export" the urls to prometheus, alertmanager for notifications, and grafana for nice gui dashboards (and maybe also notifications).

I found this other example below from this repo https://github.com/prometheus/alertmanager/issues/2559, but it is neither working.

vmalert: executes a list of the given alerting or recording rules against configured data sources. For sending alerting notifications vmalert relies on configured Alertmanager. Recording rules results are persisted via remote write protocol. vmalert is heavily inspired by Prometheus implementation and aims to be compatible with its syntax

Yes, I believe you can do all of what you're looking for without a UI. The alertmanager api has the ability to register receivers as well as to poll for alerts, silence them, etc: https://github.com/prometheus/alertmanager/blob/main/api/v2/openapi.yaml

I found it, unfortunately it doesn't help.

#cloud-config # environment: ${environment} runcmd: # install AWS CLI, neeeded for downloading of configuration files - | apt-get update && apt-get install unzip -y curl -Lo awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip unzip awscli.zip ./aws/install rm awscli.zip # install prometheus binary - | curl -Lo prometheus.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.33.1/prometheus-2.33.1.linux-arm64.tar.gz tar -xvf prometheus.tar.gz cp ./prometheus-2.33.1.linux-arm64/prometheus /usr/local/bin/prometheus rm -rf ./prometheus-2.33.1.linux-arm64 rm -rf prometheus.tar.gz # install alertmanager binary - | curl -Lo alertmanager.tar.gz https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-arm64.tar.gz tar -xvf alertmanager.tar.gz mv ./alertmanager-0.23.0.linux-arm64/alertmanager /usr/local/bin/alertmanager rm -rf alertmanager-0.23.0.linux-arm64 rm alertmanager.tar.gz # vait for EBS volume - | while [ ! -b $(readlink -f /dev/nvme1n1) ]; do echo "waiting for device /dev/nvme1n1" sleep 5 done # format volume blkid $(readlink -f /dev/nvme1n1) || mkfs -t ext4 $(readlink -f /dev/nvme1n1) # create a mount mkdir -p /data if ! grep "/dev/nvme1n1" /etc/fstab; then echo "/dev/nvme1n1 /data ext4 defaults,discard 0 0" >> /etc/fstab fi # mount volume mount /data # enable and start systemd services - | systemctl daemon-reload systemctl enable prepare-prometheus.service && systemctl start prepare-prometheus.service && sleep 10 systemctl enable prometheus.service && systemctl start prometheus.service systemctl enable alertmanager.service && systemctl start alertmanager.service write_files: - path: /usr/local/bin/prepare-prometheus permissions: '0744' content: | #!/bin/sh mkdir -p /etc/prometheus aws s3 cp s3://${s3_bucket}/prometheus.yaml /etc/prometheus/prometheus.yaml aws s3 cp s3://${s3_bucket}/alertmanager.yaml /etc/prometheus/alertmanager.yaml aws s3 cp s3://${s3_bucket}/prometheus.rules.yaml /etc/prometheus/prometheus.rules.yaml curl -X POST http://localhost:9090/-/reload || true - path: /etc/systemd/system/prepare-prometheus.service content: | [Unit] Description=Prepare prometheus / alertmanager configuration Wants=network-online.target After=network-online.target [Service] Type=oneshot ExecStart=/usr/local/bin/prepare-prometheus # please note data.mount in dependencies - path: /etc/systemd/system/prometheus.service content: | [Unit] Description=Prometheus Wants=network-online.target After=network-online.target data.mount prepare-prometheus.service [Service] Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yaml \ --storage.tsdb.path /data/ \ --web.enable-lifecycle \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries \ --enable-feature=remote-write-receiver [Install] WantedBy=multi-user.target - path: /etc/systemd/system/alertmanager.service content: | [Unit] Description=Alert Manager Wants=network-online.target After=network-online.target data.mount prepare-prometheus.service [Service] Type=simple ExecStart=/usr/local/bin/alertmanager \ --config.file /etc/prometheus/alertmanager.yaml \ --storage.path=/data/ [Install] WantedBy=multi-user.target

Prometheus recently added native support for time ranges in the alerting config https://github.com/prometheus/alertmanager/issues/876

Prometheus)Alertmanager: https://github.com/prometheus/alertmanager | https://prometheus.io/