opentelemetry-collector VS prometheus

But how does one monitor a Collector? The OTel Collector already emits metrics for the purposes of its own monitoring. These can then be sent to your Observability backend for monitoring.

Maybe, you are asking yourself: "But I already had instrumented my applications with vendor-specific libraries and I'm using their agents and monitoring tools, why should I change to OpenTelemetry?". The answer is: maybe you're right and I don't want to encourage you to update the way how you are doing observability in your applications, that's a hard and complex task. But, if you are starting from scratch or you are not happy with your current observability infrastructure, OpenTelemetry is the best choice, independently of the backend telemetry tool that you are using. I would like to invite you to take a look at the number of exporters available in the collector contrib section, if your backend tracing tool is not there, probably it's already using the Open Telemetry Protocol (OTLP) and you will be able to use the core collector. Otherwise, you should consider changing your backend telemetry tool or contributing to the project creating a new exporter.

To receive OTLP data, you set up the standard otlp receiver to receive data in HTTP or gRPC format. To forward traces and metrics, a batch processor was defined to accumulate data and send it every 100 milliseconds. Then set up a connection to Tempo (in otlp/tempo exporter, with a standard top exporter) and to Prometheus (in prometheus exporter, with a control exporter). A debug exporter also was added to log info on container standard I/O and see how the collector is working.

You can find more details on advanced configurations here.

> opentelemetry is basically a house of antipatterns

"Look on My Works Ye Mighty and Despair!"

https://github.com/open-telemetry/opentelemetry-collector/tr... -> https://github.com/open-telemetry/opentelemetry-collector-re... ... and then a reasonable person trying to load that mess into their head may ask 'err, what's the difference between go.opentelemetry.io/collector and github.com/open-telemetry/opentelemetry-collector-contrib?'

  $ curl -fsS go.opentelemetry.io/collector | grep go-import

open-telemetry/opentelemetry-collector: OpenTelemetry Collector (github.com)

There are 2 collector types: the core and the contrib. I have used the contrib as it features the splunk_hec exporter.

We are setting the deployment to have exactly 1 replica and setting the container CPU and memory limits according to the minimum that was checked for performance in their docs.

Lightstep ingests data in native OpenTelemetry Protocol (OTLP) format, so we will use the OTLP Exporter. The exporter can be called either otlp or follow the naming format otlp/. We could call it otlp/bob if we wanted to. We're calling our exporter otlp/ls to signal to us that we are using the OTLP exporter to send the data to Lightstep.

Then, we set up a batch processor that batches up the spans together and every 1 second sends the batch forward. In production, you would want more than 1 second, but I set this here to 1 second for instant feedback in Jaeger.

WriteRequest::timeseries is a vector (https://github.com/prometheus/prometheus/blob/main/prompb/re...) and

Developers widely use Prometheus as a system for operational monitoring and alerting for their projects. Here is a list of tools for monitoring frontend services with Prometheus.

Just to give an example of the power of Go for CLI builds, you may have already used or at least heard of Docker, Kubernetes, Prometheus, Terraform, but what do they all have in common? They all have a large part of their usability via CLI and are developed in Go 🐿.

Distributed system administrators need mechanisms and tools for monitoring individual nodes in order to analyze the system and promptly detect anomalies. Developers also need effective mechanisms for analyzing, diagnosing issues, and identifying bugs in protocol implementations. Logging, tracing, and collecting metrics are common observability techniques to allow monitoring and obtaining diagnostic information from the system; most of the explored code bases use these techniques. OpenTelemetry and Prometheus are popular open-source monitoring solutions, which are used in many of the explored code bases.

Setting up monitoring for a system, especially one involving GRPC communication, provides crucial visibility into its operations. In this guide, we walked through the steps to instrument both a GRPC server and client with Prometheus metrics, exposed those metrics via an HTTP endpoint, and visualized them using Grafana. The Docker-Compose setup simplified the deployment of both Prometheus and Grafana, ensuring a streamlined process.

Alerting and Notification: Select a tool with flexible alerting mechanisms to proactively detect anomalies or deviations from defined thresholds. Consider asking questions like "Does this tool offer customizable alerting options and support notification channels that suit our team's communication preferences?" A tool like Prometheus provides robust alerting capabilities.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit. It is designed to monitor highly dynamic containerized systems, making it an excellent choice for monitoring Docker containers and Kubernetes clusters.

wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz

Prometheus: Open-source monitoring system. Often used together with Grafana.