Helm-Chart-Boilerplates VS thanos

I would point you to what I call the "Universal Helm Charts" and some examples of how to use them.

Shameless Plug: Here's one of my dashboards I made for Ingress-Nginx, which is my recommended border router/gateway into all the services. It adds deep robust metrics and configurability, and if you've got years of experience with Nginx also, it allows you rich complex customization via nginx's configuration structure via kubernetes annotations. Besides that I have open-source helm charts which are easy to use, boilerplates showing how to use them, a volume autoscaler to automatically resize your disks as they get full, and a blog where I share various of my experience which is a companion blog to my upcoming book of the same name. Hope this helps! Feel free to ask if you have any further questions.

Here is an example of a repo that uses an sub-chart: https://github.com/DevOps-Nirvana/Helm-Chart-Boilerplates/tree/master/boilerplate-apache-with-configmap-template/deployment

Use multi-values files with helm ALWAYS. Allowing an env-specific overlay to tweak your default values files. See: https://github.com/DevOps-Nirvana/Helm-Chart-Boilerplates/tree/master/boilerplate-echoserver/deployment/boilerplate-echoserver

Helm charts - https://github.com/DevOps-Nirvana/Universal-Kubernetes-Helm-Charts Example using helm charts as sub charts - https://github.com/DevOps-Nirvana/Helm-Chart-Boilerplates/tree/master/boilerplate-echoserver

Helm Chart Boilerplates are examples of usage of the above Universal Helm Charts to help people understand how to use them more, a stop-gap until I add more documentation

Helm Chart Usage Boilerplates (Examples of using these helm chart)

To help you on deploying your service, I've created open source generic/universal Helm Charts to make it easy to do the above. Here are the Universal Helm Charts and here's some boilerplate examples of using them. These built-in have support for HPAs, services, ingresses, etc, making it as easy as autoscaling.enable: true I haven't gotten around to documenting the helm charts yet, but there's lots of comments in the values.yaml file explaining everything.

Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

Sounds like you want something like Thanos

Yes, but also no. The Prometheus ecosystem already has two FOSS time-series databases that are complementary to Prometheus itself. Thanos and Mimir. Not to mention M3db, developed at Uber, and Cortex, then ancestor of Mimir. There's a bunch of others I won't mention as it would take too long.

Long term storage all depends on your needs and sophistication. I use Thanos for our system since it has an extremely flexible scaling system. But there is also Grafana Mimir. They're both similar in that they use Prometheus TSDB format as part of the underlying storage. One nice Thanos advantage is that it does do downsampling in addition to being able to store raw metric data for a long time. It will auto-select downsampled data to make requests faster.

You can aggregate all your clusters Prometheus metrics together with a wonderful tool called Thanos. This will allow you to use just a single Grafana instance against Thanos and using a label select which cluster you wish to see metrics from. The downside of this, is that none of the Grafana dashboards from the internet will work as-is. You'll need to customize all of them for Thanos support. The other downside is, you have a single point of failure, and (see next item) you can't customize who can access what in regards to your dev vs production data/metrics/access.

Depending on how you want to set things up, you can use Thanos or Mimir to create the single-pane-of-glass view of your data.

You could look at something like Thanos and store your data in S3: https://thanos.io/