Those of you using prometheus as part of your observability stack, what approach did you take to scaling to scrape 25+ clusters, and why? Is Thanos the answer to my problems?

• thanos

  66 12,585 9.6 Go

  Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

  

I understand that Thanos (https://github.com/thanos-io/thanos) was built with the idea of improving prom's scalability and availability , but would love to hear from others that have tried various approaches to try to solve this.

• agent

  21 1,545 9.9 Go

  Vendor-neutral programmable observability pipelines. (by grafana)

  

Furthermore, would recommend Grafana Agent OR Prometheus Agent in this case since you probably don't need the Prometheus UI in each Cluster as well as the Alerting stuff that is inside Prometheus. (Mimir will do the ruling stuff for you). Grafana Agent also has an Operator mode if you want to use ServiceMonitor and PodMonitor CustomResources.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project