thanos VS pritunl-k8s-tf-do

Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

Sounds like you want something like Thanos

Yes, but also no. The Prometheus ecosystem already has two FOSS time-series databases that are complementary to Prometheus itself. Thanos and Mimir. Not to mention M3db, developed at Uber, and Cortex, then ancestor of Mimir. There's a bunch of others I won't mention as it would take too long.

Long term storage all depends on your needs and sophistication. I use Thanos for our system since it has an extremely flexible scaling system. But there is also Grafana Mimir. They're both similar in that they use Prometheus TSDB format as part of the underlying storage. One nice Thanos advantage is that it does do downsampling in addition to being able to store raw metric data for a long time. It will auto-select downsampled data to make requests faster.

You can aggregate all your clusters Prometheus metrics together with a wonderful tool called Thanos. This will allow you to use just a single Grafana instance against Thanos and using a label select which cluster you wish to see metrics from. The downside of this, is that none of the Grafana dashboards from the internet will work as-is. You'll need to customize all of them for Thanos support. The other downside is, you have a single point of failure, and (see next item) you can't customize who can access what in regards to your dev vs production data/metrics/access.

Depending on how you want to set things up, you can use Thanos or Mimir to create the single-pane-of-glass view of your data.

You could look at something like Thanos and store your data in S3: https://thanos.io/

I have a full example here of a working GHA pipeline that deploys terraform infrastructure. This deploys atlantis, which can then be used to deploy Pritunl VPN. Works almost perfectly except for the fact that helm yaml encoded sensitive values are revealed on terraform destroy, so I simply don't have anything sensitive encoded in yaml.

Hey now, I've already got a homelab that launches a k8s cluster and installs Pritunl VPN for coffee shop wifi. I do at least understand that tools need a valid use case before being applied, although admittedly installing it on k8s vs just using Nomad or something is more RDD than not for this one.

I'm looking for a company that is willing to judge me based off of past experience and previous projects I've already completed. So ideally no LeetCode, and no more take homes unless you want me to post them publicly. Ideally looking to come in at $160K+ for mid level or $200K+ for senior, depending on how much of a match there is between me and the position, and a sign-on bonus would be incredible. Hopefully I have enough of a reputation at this point where you've seen my posts and comments in /r/devops and already know what I'm capable of, but if not, I'm happy to chat about previous projects in-depth and go over what I've worked on. A code review as part of the interview process would be absolutely stellar. This is what I've been working on recently as well as my "homelab" if you re looking for some specifics. DM or chat with your work email for an official resume

Already done, but good call on learning how to create conditionals. I will look into it!

Playing around with grafana/loki/prometheus all via Terraform, GitHub Actions, and Atlantis in a public repo.

I set up a whole pipeline to install and configure Pritunl VPN on DigitalOcean and it only costs me like $60/month for a 3 node cluster.

https://github.com/autotune/pritunl-k8s-tf-do/blob/master/README.md is the repo. The README should answer any questions about how the pipeline works but the end result is a pritunl webgui listening on port 80 with an ingress route for https, a SERVICE load balancer that listens for VPN connections, and the ability to connect to said service load balancer over Pritunl VPN client. Note this is missing a few things, for one you can only have a replica set of 1 in the deployment. I need to figure out how to add HA with the "enterprise" edition at $70/month extra (still relatively cheap for what you get!). But for personal use it should suffice. Also, I tried using an ingress for the vpn itself but can't get it working, so stuck with SVC load balaner instead, which works fine. Any suggestions here would be appreciated!

All I know for sure is the one cert I was using with letsencrypt kept failing to renew. I just tried it with zerossl since the sign up page cert was finally renewed last night and people have generally been happy with them outside this little incident and seems to actually be working as expected. The helm release I am using is linked to via tf here and the ingress rules are here.