alertmanager-status VS covid_status

I dunno, I don't really mind self-hosting monitoring infrastructure. I basically pay for a website uptime checker to check that Alertmanager is working. If Alertmanager is down, obviously you have to manually check to see what else is down, but it doesn't fail open.

I wrote a little glue to make this straightforward for anyone else who uses Prometheus/Alertmanager: https://github.com/jrockway/alertmanager-status This ensures that the website check checks the health of the whole alerting pipeline; Prometheus has an always firing alert, Alertmanager is set to send that alert to alertmanager-status, and alertmanager-status starts failing its external health check if it isn't seeing that alert firing at the configured interval. If one of [Prometheus, Alertmanager, alertmanager-status] fails, then your website health check fails.

It is quite awkward that the output of "working" and "completely broken" alerting systems have the same visible effect -- no alerts.

For Prometheus users, I wrote alertmanager-status to let a third-party "website up?" monitoring server check your alertmanager: https://github.com/jrockway/alertmanager-status

(I also wrote one of the main Google Fiber monitoring systems back when I was at Google. We spent quite a bit of time on monitoring monitoring, because whenever there was an actual incident people would ask us "is this real, or just the monitoring system being down?" Previous monitoring systems were flaky so people were kind of conditioned to ignore the improved system -- so we had to have a lot of dashboards to show them that there was really an ongoing issue.)

Many things!

jsso2: Identity provider and authenticating proxy for your non-enterprise use cases. WebAuthn only, no passwords! I was tired of typing a password for things like Grafana and PGAdmin, and IP whitelisting my home Internet for things that didn't have built-in authentication. https://github.com/jrockway/jsso2

If I were starting from 0 today, I'd just use Dex and Envoy's built-in OAuth support. OAuth is overly complicated, requiring a bunch of configuration for each app, and a ton of code in each app... but it won. So use that.

jlog: I read a lot of log files in my day-to-day work and really like the idea of structured logs, but found them hard to read. jlog translates timestamps to my local time zone, lets me query them with jq, etc.: https://github.com/jrockway/json-logs Can't live without it, I use it many times every day, and have even convinced other people to use it without writing any documentation. (There are binary releases and a --help though!)

"kubectl jq": I wanted to play with writing Kubernetes plugins, so I made one that is just "kubectl get x -o json | jq". I use it pretty regularly, but the Kubernetes client machinery doesn't give you autocompletion for free, so it's pretty painful to use. When they fix that, I plan to write more kubernetes extensions (including one that invokes jlog on the logs, saving a pipe ;) https://github.com/jrockway/kubectl-jq

alertmanager-status: How do you know if your Prometheus/Alertmanager is working? If it breaks, it won't be sending you an alert, after all. https://github.com/jrockway/alertmanager-status

ekglue: The good parts of Istio, written by someone who read the xDS spec :P https://github.comjrockway/ekglue

For my day job, I work on Pachyderm Hub, which you should totally use if you want to run production-quality data science workloads (data provenance, reproducibility, etc.): https://hub.pachyderm.com/ I could write a lot about it, but basically... we have customers that want to use Pachyderm, but the complexity of Kubernetes stands in their way. How do you store logs? How do you monitor things? How do you give your coworkers access? We solve those problems by letting you click a button in a web UI. (As for why you'd want to use Pachyderm: https://www.pachyderm.com/use-cases/)

I hacked together a Wayland (Sway/Waybar) covid-status indicator which extracts covid19 case data filtered by country (new cases / new deaths etc) from "ourworldindata.org" and displays it either in waybar or on the cli.

Apart from running (with on-click actions in Waybar) it can also be run on the command line. Output is either text, json, pdf, jp or a mako/dunst notification.

For screen-recording/screenshots see the unixporn post or my gitlab...

/r/unixporn: https://www.reddit.com/r/unixporn/comments/koazs0/sway_covid...

gitlab: https://gitlab.com/jbauernberger/covid_status/

source for covid_status indicator module: https://gitlab.com/jbauernberger/covid_status