berglas VS thanos

Secret Manager: secure storage for sensitive data e.g passwords.

Google Secret Manager

API keys are easy to "leak" or compromise, so best to not only use the restrictions presented to you when you create them but physically protect them as well. Don't code them in plain-text, don't check them into GitHub, etc. Store them in a secure database or use a service like GCP Secret Manager.

I've been looking at Google Secret Manager which sounds promising but I've not been able to find any examples or tutorials that help with the actual practical details of best practice or getting this working. I'm currently reading about Cloud Functions which also sound promising but again, I'm just going deeper and deeper into GCP without feeling like I'm gaining any useful insights.

In this post, I described how to build secure GitHub Actions workflows by pull_request_target event instead of pull_request event. Using pull_request_target, you can prevent malicious codes from being executed in CI. And by managing secrets in secrets management services such as AWS Secrets Manager and Google Secret Manager and access them via OIDC, you can restrict the access to secrets securely. To migrate pull_request to pull_request_target, several modifications are needed. And pull_request_target has a drawback that it's difficult to test changes of workflows, so it's good to introduce pull_request_target to repositories that require strong permissions in CI. For example, a Terraform Monorepo tends to require strong permissions for CI, so it's good to introduce pull_request_target to it.

If you want to make these secrets more secure and get versioning and access logs for them, you may want to switch to Secret Manager later on. They can still be exposed as environment variables to your code. It's a little more setup work, so start with the simple approach at the top.

https://cloud.google.com/secret-manager https://aws.amazon.com/secrets-manager/

Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

Sounds like you want something like Thanos

Yes, but also no. The Prometheus ecosystem already has two FOSS time-series databases that are complementary to Prometheus itself. Thanos and Mimir. Not to mention M3db, developed at Uber, and Cortex, then ancestor of Mimir. There's a bunch of others I won't mention as it would take too long.

Long term storage all depends on your needs and sophistication. I use Thanos for our system since it has an extremely flexible scaling system. But there is also Grafana Mimir. They're both similar in that they use Prometheus TSDB format as part of the underlying storage. One nice Thanos advantage is that it does do downsampling in addition to being able to store raw metric data for a long time. It will auto-select downsampled data to make requests faster.

You can aggregate all your clusters Prometheus metrics together with a wonderful tool called Thanos. This will allow you to use just a single Grafana instance against Thanos and using a label select which cluster you wish to see metrics from. The downside of this, is that none of the Grafana dashboards from the internet will work as-is. You'll need to customize all of them for Thanos support. The other downside is, you have a single point of failure, and (see next item) you can't customize who can access what in regards to your dev vs production data/metrics/access.

Depending on how you want to set things up, you can use Thanos or Mimir to create the single-pane-of-glass view of your data.

You could look at something like Thanos and store your data in S3: https://thanos.io/