Grafana VS node_exporter

Monitoring application logs is a crucial aspect of the software development and deployment lifecycle. In this post, we'll delve into the process of observing logs generated by Docker container applications operating within HashiCorp Nomad. With the aid of Grafana, Vector, and Loki, we'll explore effective strategies for log analysis and visualization, enhancing visibility and troubleshooting capabilities within your Nomad environment.

To help us visualize these scenarios, we'll build a Grafana Dashboard so we can follow along.

Visualization and Analysis: Choose a tool with intuitive and customizable dashboards, charts, and visualizations. A question to ask is, "Are the visualization features of this tool user-friendly and adaptable to our team's specific needs?" Tools like Grafana and Kibana provide powerful visualization capabilities.

Prometheus: Open-source monitoring system. Often used together with Grafana.

Grafana

Yes I'm aware that Grafana is open source but the method I used to find the API endpoints is far quicker than digging through hundreds of files in a codebase I'm not familiar with.

So, you will add one last container to allow us to visualize this data: Grafana, an open-source analytics and visualization platform that allows us to see traces and metrics simply. You can set Grafana to read data from both Tempo and Prometheus by setting them as datastores with the following grafana.datasource.yaml config file:

In example above, we use 2 additional parameters: code (HTTP response code) and page (page identifier), which provide detailed statistics. For example, you can build such graphs in Grafana:

Robin switched to the Grafana dashboard tab, and sure enough, the 5xx volume on web service was rising. It had not hit the critical alert thresholds yet, but customers had already started noticing.

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz $ tar -xzvf node_exporter-1.7.0.linux-amd64.tar.gz

Node Exporter

The Prometheus node_exporter can provide this information, doesn't require root. You could run it as a systemd user unit if you don't have root.

I personally find it best to learn through experimentation. Start with reading a bit about Prometheus and Grafana through their docs, and then familiarise yourself with setting up a local Prometheus + Grafana instance either locally or with docker using docker-compose, along with something to generate /metrics endpoint for Prometheus to scrape from such as a custom Prometheus exporter in Python or using Node Exporter.

Your best bet might be to fork node_exporter to get you more verbose socket stats: https://github.com/prometheus/node_exporter/blob/master/collector/sockstat_linux.go

I use Grafana + Prometheus + Node Exporter.

Crazy idea, can't NX devices run Docker? I wonder if the node_exporter would work.

Kubernetes itself will not notify you, the way I've seen people do this, is to use something like kube-state-metrics or node_exporter, export that to Prometheus (or preferrably VictoriaMetrics because Prometheus is terrible IMO), and then setup alarms on that with alertmanager or equivalent, or just look at dashboards regularly with Grafana. Realistically I recommend only setting alerts on disk usage and application/database latency. CPU and memory utilization isn't a great metric to alert on a lot of the time.

You may setup node exporter and collect metrics with prometheus for example. Its not quite "simple" way, but still you may find it useful.