alertmanager VS zipkin

- Alert Manager

You could use prometheus as a monitoring tool, blackbox_exporter to "export" the urls to prometheus, alertmanager for notifications, and grafana for nice gui dashboards (and maybe also notifications).

I found this other example below from this repo https://github.com/prometheus/alertmanager/issues/2559, but it is neither working.

vmalert: executes a list of the given alerting or recording rules against configured data sources. For sending alerting notifications vmalert relies on configured Alertmanager. Recording rules results are persisted via remote write protocol. vmalert is heavily inspired by Prometheus implementation and aims to be compatible with its syntax

Yes, I believe you can do all of what you're looking for without a UI. The alertmanager api has the ability to register receivers as well as to poll for alerts, silence them, etc: https://github.com/prometheus/alertmanager/blob/main/api/v2/openapi.yaml

I found it, unfortunately it doesn't help.

#cloud-config # environment: ${environment} runcmd: # install AWS CLI, neeeded for downloading of configuration files - | apt-get update && apt-get install unzip -y curl -Lo awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip unzip awscli.zip ./aws/install rm awscli.zip # install prometheus binary - | curl -Lo prometheus.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.33.1/prometheus-2.33.1.linux-arm64.tar.gz tar -xvf prometheus.tar.gz cp ./prometheus-2.33.1.linux-arm64/prometheus /usr/local/bin/prometheus rm -rf ./prometheus-2.33.1.linux-arm64 rm -rf prometheus.tar.gz # install alertmanager binary - | curl -Lo alertmanager.tar.gz https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-arm64.tar.gz tar -xvf alertmanager.tar.gz mv ./alertmanager-0.23.0.linux-arm64/alertmanager /usr/local/bin/alertmanager rm -rf alertmanager-0.23.0.linux-arm64 rm alertmanager.tar.gz # vait for EBS volume - | while [ ! -b $(readlink -f /dev/nvme1n1) ]; do echo "waiting for device /dev/nvme1n1" sleep 5 done # format volume blkid $(readlink -f /dev/nvme1n1) || mkfs -t ext4 $(readlink -f /dev/nvme1n1) # create a mount mkdir -p /data if ! grep "/dev/nvme1n1" /etc/fstab; then echo "/dev/nvme1n1 /data ext4 defaults,discard 0 0" >> /etc/fstab fi # mount volume mount /data # enable and start systemd services - | systemctl daemon-reload systemctl enable prepare-prometheus.service && systemctl start prepare-prometheus.service && sleep 10 systemctl enable prometheus.service && systemctl start prometheus.service systemctl enable alertmanager.service && systemctl start alertmanager.service write_files: - path: /usr/local/bin/prepare-prometheus permissions: '0744' content: | #!/bin/sh mkdir -p /etc/prometheus aws s3 cp s3://${s3_bucket}/prometheus.yaml /etc/prometheus/prometheus.yaml aws s3 cp s3://${s3_bucket}/alertmanager.yaml /etc/prometheus/alertmanager.yaml aws s3 cp s3://${s3_bucket}/prometheus.rules.yaml /etc/prometheus/prometheus.rules.yaml curl -X POST http://localhost:9090/-/reload || true - path: /etc/systemd/system/prepare-prometheus.service content: | [Unit] Description=Prepare prometheus / alertmanager configuration Wants=network-online.target After=network-online.target [Service] Type=oneshot ExecStart=/usr/local/bin/prepare-prometheus # please note data.mount in dependencies - path: /etc/systemd/system/prometheus.service content: | [Unit] Description=Prometheus Wants=network-online.target After=network-online.target data.mount prepare-prometheus.service [Service] Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yaml \ --storage.tsdb.path /data/ \ --web.enable-lifecycle \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries \ --enable-feature=remote-write-receiver [Install] WantedBy=multi-user.target - path: /etc/systemd/system/alertmanager.service content: | [Unit] Description=Alert Manager Wants=network-online.target After=network-online.target data.mount prepare-prometheus.service [Service] Type=simple ExecStart=/usr/local/bin/alertmanager \ --config.file /etc/prometheus/alertmanager.yaml \ --storage.path=/data/ [Install] WantedBy=multi-user.target

Prometheus recently added native support for time ranges in the alerting config https://github.com/prometheus/alertmanager/issues/876

Prometheus)Alertmanager: https://github.com/prometheus/alertmanager | https://prometheus.io/

When choosing distributed tracing tools, considerations include your technology stack, business requirements, and monitoring complexity. Zipkin, SkyWalking, and OpenTelemetry are popular distributed tracing solutions, each with its unique features.

From the perspective of the realization of GraphQL infrastructure, the interesting direction is "Finding". How to find the problem? How to find the bottleneck of the system? Distributed Tracing System (DTS) will help answer this question. Distributed tracing is a method of observing requests as they propagate through distributed environments. In our scenario, we have dozens of subgraphs, gateway, and transport layer through which the request goes. We have several tools that can be used to detect the whole lifecycle of the request through the system, e.g. Jaeger, Zipkin or solutions that provided DTS as a part of the solution NewRelic.

Zipkin is a distributed tracing system used for tracking and analyzing how requests move through complex systems, especially in setups with many interconnected services, known as microservices.

Distributed Tracing: Middleware for distributed tracing like Jaeger and Zipkin helps monitor and trace requests as they flow through multiple microservices, aiding in debugging, performance optimization, and understanding the system's behavior.

For microservice tracing, you might want to look at Zipkin [0], or OpenTelemetry [1]

[0] https://zipkin.io/

I’ve not used a self-hosted solution before, but here’s one I found. https://zipkin.io/

> IMO the reason these vendors can and do charge so much is not because telemetry software is hard.

I always saw it as "they are charging for their polished UI/experience"

The UI of https://zipkin.io/ versus DataDog is kind of... not really in the same ballpark?

There are the zipkin https://zipkin.io/ and jaeger https://www.jaegertracing.io/ packages/components you can use both have quickstarts if you consider that to be a beginner's guide.

Zipkin, which was developed by Twitter, is an open source tool for distributed tracing that can also be used to troubleshoot latency issues in your application. While Zipkin is Java-based, py_zipkin is an implementation for Python.