prometheus VS Collectd

WriteRequest::timeseries is a vector (https://github.com/prometheus/prometheus/blob/main/prompb/re...) and

Developers widely use Prometheus as a system for operational monitoring and alerting for their projects. Here is a list of tools for monitoring frontend services with Prometheus.

Just to give an example of the power of Go for CLI builds, you may have already used or at least heard of Docker, Kubernetes, Prometheus, Terraform, but what do they all have in common? They all have a large part of their usability via CLI and are developed in Go 🐿.

Distributed system administrators need mechanisms and tools for monitoring individual nodes in order to analyze the system and promptly detect anomalies. Developers also need effective mechanisms for analyzing, diagnosing issues, and identifying bugs in protocol implementations. Logging, tracing, and collecting metrics are common observability techniques to allow monitoring and obtaining diagnostic information from the system; most of the explored code bases use these techniques. OpenTelemetry and Prometheus are popular open-source monitoring solutions, which are used in many of the explored code bases.

Setting up monitoring for a system, especially one involving GRPC communication, provides crucial visibility into its operations. In this guide, we walked through the steps to instrument both a GRPC server and client with Prometheus metrics, exposed those metrics via an HTTP endpoint, and visualized them using Grafana. The Docker-Compose setup simplified the deployment of both Prometheus and Grafana, ensuring a streamlined process.

Alerting and Notification: Select a tool with flexible alerting mechanisms to proactively detect anomalies or deviations from defined thresholds. Consider asking questions like "Does this tool offer customizable alerting options and support notification channels that suit our team's communication preferences?" A tool like Prometheus provides robust alerting capabilities.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit. It is designed to monitor highly dynamic containerized systems, making it an excellent choice for monitoring Docker containers and Kubernetes clusters.

wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz

Prometheus: Open-source monitoring system. Often used together with Grafana.

https://collectd.org/ does the gathering (and writing to RRDTool database, if you so desire) part very well. Many plugins, easy to add more (just return one line of text)

Still need RRD viewere but that's not a huge stack

And it scales all the way to hundreds of hosts, as on top of network send/receive of stats it supports few other write formats aside from just RRD files.

Then you will have same problems but now you can bother manufacturer about it!

Also unless there is something horribly wrong about how often data is written, that SSD should run for ages.

We ran (for a test) consumer SSDs in busy ES cluster and they still lasted like 2 years just fine

The whole setup was a bit of overcomplicated too. RAID10 with 5+1 or 7+1 (yes Linux can do 7 drive RAID10) with hotspare woud've been entirely fine, easier, and most likely faster. You need backups anyway so ZFS doesn't give you much here, just extra CPU usage

Either way, monitoring wait per drive (easy way is to just plug collectd [1] into your monitoring stack, it is light and can monitor A TON of different metrics)

* [1]https://collectd.org/

Collectd pulls metrics from the OS, applications, logfiles and external devices for use in monitoring systems, finding performance bottlenecks and capacity planning. hombre_sabio explains, "Collectd is a tiny daemon that gathers information from a system. It enables mechanisms to collect and observe the values in different techniques. It is an open-source monitoring tool to retrieve and manage SNMP master agents."

Setting up php7.4-fpm (7.4.25-1+deb11u1) ... Job for php7.4-fpm.service failed because a fatal signal was delivered to the co ntrol process. See "systemctl status php7.4-fpm.service" and "journalctl -xe" for details. invoke-rc.d: initscript php7.4-fpm, action "start" failed. ● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor pre set: enabled) Active: failed (Result: signal) since Mon 2021-12-27 23:53:51 GMT; 215ms ag o Docs: man:php-fpm7.4(8) Process: 2755 ExecStart=/usr/sbin/php-fpm7.4 --nodaemonize --fpm-config /etc /php/7.4/fpm/php-fpm.conf (code=killed, signal=ILL) Process: 2756 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/ph p/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCES S) Main PID: 2755 (code=killed, signal=ILL) CPU: 281ms Dec 27 23:53:51 raspberrypi systemd[1]: Starting The PHP 7.4 FastCGI Process Man ager... Dec 27 23:53:51 raspberrypi systemd[1]: php7.4-fpm.service: Main process exited, code=killed, status=4/ILL Dec 27 23:53:51 raspberrypi systemd[1]: php7.4-fpm.service: Failed with result ' signal'. Dec 27 23:53:51 raspberrypi systemd[1]: Failed to start The PHP 7.4 FastCGI Proc ess Manager. dpkg: error processing package php7.4-fpm (--configure): installed php7.4-fpm package post-installation script subprocess returned error exit status 1 Setting up collectd (5.12.0-7.1) ... Job for collectd.service failed because a fatal signal was delivered to the cont rol process. See "systemctl status collectd.service" and "journalctl -xe" for details. invoke-rc.d: initscript collectd, action "restart" failed. ● collectd.service - Statistics collection and monitoring daemon Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor prese t: enabled) Active: activating (auto-restart) (Result: signal) since Mon 2021-12-27 23: 53:52 GMT; 200ms ago Docs: man:collectd(1) man:collectd.conf(5) https://collectd.org Process: 2768 ExecStartPre=/usr/sbin/collectd -t (code=killed, signal=SEGV) CPU: 24ms dpkg: error processing package collectd (--configure): installed collectd package post-installation script subprocess returned error e xit status 1 dpkg: dependency problems prevent configuration of openmediavault: openmediavault depends on collectd; however: Package collectd is not configured yet. dpkg: error processing package openmediavault (--configure): dependency problems - leaving unconfigured Errors were encountered while processing: php7.4-fpm collectd openmediavault E: Sub-process /usr/bin/dpkg returned an error code (1)

For metrics storage I'm using a Graphite database and the graph UI itself is Grafana. To get these I'm using the Debian repos they supply with mostly off-the-shelf configs. For collecting metrics from the Pi to send to Graphite I use collectd. It has a lot of off-the-shelf plugins you can use to grab metrics like CPU usage & load average, network in/out, memory stats etc. The Minecraft-specific stuff you can get from configuring collectd plugins as well, like the tick lag graph I use the "tail" plugin to follow and parse the server log.

For monitoring, personally I use collectd and Collectd Graph Panel (sadly the latter is abandoned, but it still works fine)