AKS VS thanos

Ah yep, you're right. Just found someone asking the same thing from two days ago in the Azure AKS GitHub https://github.com/Azure/AKS/issues/3689. It should definitely be pulling the Ubuntu 22.04 fips compliant image if it were available.

Just did a search not sure related to your problem https://github.com/Azure/AKS/issues/85

As I could gather additional information which indicates some problems with the Monitoring Agent (high cpu on ama pods, no metrics delivered during CPU pressure, ama pods emitting logs), I opened a ticket on GitHub (https://github.com/Azure/AKS/issues/3469). Hopefully this helps to resolve the issue.

While I'm glad I saw this here, it's not one of our primary support channels. In the future, you can also raise an issue in our GitHub repository or file a support request.

You're dependent on the cloud provider for security settings in a lot of cases and they don't always get it right (e.g. AKS still doesn't have Node Authorization enabled! https://github.com/Azure/AKS/issues/3004)

For ASK even v1.22 got out of support yesterday 04 of December. https://github.com/Azure/AKS/releases/tag/2022-11-27

The Workload Identity add-on for AKS is still in progress. AKS has a public roadmap where you can find this kind of information.

Reference: https://github.com/Azure/AKS

For Thanos receive and query components to be available outside the cluster and secured with TLS, we will need ingress-nginx and cert-manager. For ingress, deploy the Helm chart using the following command, to account for this issue with AKS clusters >1.23:

Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

Sounds like you want something like Thanos

Yes, but also no. The Prometheus ecosystem already has two FOSS time-series databases that are complementary to Prometheus itself. Thanos and Mimir. Not to mention M3db, developed at Uber, and Cortex, then ancestor of Mimir. There's a bunch of others I won't mention as it would take too long.

Long term storage all depends on your needs and sophistication. I use Thanos for our system since it has an extremely flexible scaling system. But there is also Grafana Mimir. They're both similar in that they use Prometheus TSDB format as part of the underlying storage. One nice Thanos advantage is that it does do downsampling in addition to being able to store raw metric data for a long time. It will auto-select downsampled data to make requests faster.

You can aggregate all your clusters Prometheus metrics together with a wonderful tool called Thanos. This will allow you to use just a single Grafana instance against Thanos and using a label select which cluster you wish to see metrics from. The downside of this, is that none of the Grafana dashboards from the internet will work as-is. You'll need to customize all of them for Thanos support. The other downside is, you have a single point of failure, and (see next item) you can't customize who can access what in regards to your dev vs production data/metrics/access.

Depending on how you want to set things up, you can use Thanos or Mimir to create the single-pane-of-glass view of your data.

You could look at something like Thanos and store your data in S3: https://thanos.io/