thanos VS VictoriaMetrics

Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

Sounds like you want something like Thanos

Yes, but also no. The Prometheus ecosystem already has two FOSS time-series databases that are complementary to Prometheus itself. Thanos and Mimir. Not to mention M3db, developed at Uber, and Cortex, then ancestor of Mimir. There's a bunch of others I won't mention as it would take too long.

Long term storage all depends on your needs and sophistication. I use Thanos for our system since it has an extremely flexible scaling system. But there is also Grafana Mimir. They're both similar in that they use Prometheus TSDB format as part of the underlying storage. One nice Thanos advantage is that it does do downsampling in addition to being able to store raw metric data for a long time. It will auto-select downsampled data to make requests faster.

You can aggregate all your clusters Prometheus metrics together with a wonderful tool called Thanos. This will allow you to use just a single Grafana instance against Thanos and using a label select which cluster you wish to see metrics from. The downside of this, is that none of the Grafana dashboards from the internet will work as-is. You'll need to customize all of them for Thanos support. The other downside is, you have a single point of failure, and (see next item) you can't customize who can access what in regards to your dev vs production data/metrics/access.

Depending on how you want to set things up, you can use Thanos or Mimir to create the single-pane-of-glass view of your data.

You could look at something like Thanos and store your data in S3: https://thanos.io/

VictoriaMetrics CTO here.

The referred library is the official OpenTelemetry package for reading metrics in Go language [1] - more details are available at [2].

Note that we at VictoriaMetrics like the idea of unified observability standard like OpenTelemetry. The issue is in the current otel implementation. It is too bloated and very inefficient. This contradicts to our experience with observability cases, which need very optimized format for metrics' transfer in order to reduce costs on CPU and network traffic needed to transfer and process these metrics.

VictoriaMetrics continues investing in OpenTelemetry by providing integration docs [3] and improving the existing functionality for otel metrics' ingestion [4].

[1] https://github.com/open-telemetry/opentelemetry-proto-go

[2] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570...

[3] https://docs.victoriametrics.com/guides/getting-started-with...

[4] https://github.com/VictoriaMetrics/VictoriaMetrics/issues/60...

Victoria Metrics

VictoriaMetrics is primarily a time-series database designed for efficiently storing and querying time-series data. It is often used as a back-end data store for time-series data generated by monitoring systems like Prometheus. VictoriaMetrics excels at handling large volumes of time-series data, offering efficient storage and query capabilities.

Not sure I follow since there are very competitive tools written in Go such as https://victoriametrics.com for an example in this space.

Did you try VictoriaMetrics [1] and vmagent [2]? It is a single self-contained binary without external dependencies. It requires relatively low amounts of CPU, RAM, disk space and disk IO, and it runs on ARM.

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/

[2] https://docs.victoriametrics.com/vmagent.html

https://github.com/VictoriaMetrics/VictoriaMetrics#cardinali...

If I understanding correctly, it deal with high cardinality by dropping data, the operators need to monitor for this and adjust their data to lower the cardinality.

VictoriaMetrics

You shouldn't unless you want to use the new open source standard for telemetry. You won't benefit from simplicity or performance improvements. It would be quite the opposite. You can check what is the actual cost of open telemetry adoption here [0]

But if you ever decide to go this path - VictoriaMetrics supports OpenTelemetry protocol for metrics [1]

[0] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570

[1] https://docs.victoriametrics.com/Single-server-VictoriaMetri...