cloudprober VS sloth

Do you mean that your service doesn't have enough traffic all the time? Then you can (and should) use synthetic clients to send requests to your endpoint. They provide both a minimal amount of traffic all day round and also can report on responses they get and improve your coverage. Example project: https://github.com/google/cloudprober

If your service might at times fall to almost zero requests outside business hours, having a synthetic client is a must. You can use Blackbox exporter as mentioned by u/SuperQue or CloudProber, both work well for simple cases (one step site check or API call), for anything more complicated (multi step scenarios) you are better off scripting it.

Relatively easy to achieve: through variety of available opensource projects like cloudprober or blackbox exporter (if your test case is straight forward) or custom made programs out of bash, python, golang (if your test case is more complex).

It’s a CLI that allows developers to embed SLO annotation into GO code as comments and generate Prometheus alert groups when paired with Sloth, https://github.com/slok/sloth.

SLOTH: https://github.com/slok/sloth

Note: Although it’s outside of the scope of this post to dig deep into this topic, in case you’re curious, you can check out what an OpenSLO YAML definition looks like here.

The actual hard part is standardizing all teams on SLI/SLO-based thinking. For that we're looking at tools like Sloth.

Have a look at sloth (https://github.com/slok/sloth) which will help you generate SLOs and error budgets given a PromQL query. This might be easier than trying to calculate it yourself. Plus, it's "metrics as code" and OpenSLO spec compliant.

If you are in k8s and use Prometheus you could take a look at sloth: https://github.com/slok/sloth which can either generate the rules/alerts for you, or can run as an operator and allows you to write SLOs as k8s kinds.

Check out https://github.com/slok/sloth