kibana VS prometheus

As you might have guessed, I spend a lot of time thinking about application security - almost every day, in fact. At my day job, I'm constantly pondering how to enhance Kibana's security in a scalable manner without overburdening my already hardworking team. Outside of work, I'm equally dedicated to making Secutils.dev even more valuable to fellow engineers looking for better security tools.

These newsletters are among the best sources to stay up-to-date with the latest happenings in JavaScript, web development, and Node.js. They conveniently categorize content into sections like new releases, articles & tutorials, and code & tools. Since I use JavaScript/TypeScript and Node.js extensively, both in my day job and for Secutils.dev, I have to stay informed about developments in these essential tools. Usually, I quickly scan through the newsletter and focus only on the items that grab my attention — it doesn't consume much time but keeps me well-informed.

This is the second part of my reflection sparked by the recent “2023 State of Open Source Security” report from Snyk. It got me thinking about the price we pay for false positives in software security. In my previous post, “The Cost of False Positives in Software Security, Part 1: Small Applications”, I talked about how true and false positives affect smaller applications like Secutils.dev. Now, I want to take the same idea and apply it to a much larger software that’s a big part of my daily work: Kibana.

False positives in security are something that really bothers me, as I happen to work on security for both large applications like Kibana, with hundreds of contributors, and smaller ones like Secutils.dev, where I'm the sole developer.

The increasing complexity of modern systems led to the rise of AIOps (Artificial Intelligence for IT Operations) and observability practices. AIOps leveraged machine learning algorithms to automate problem detection, analysis, and resolution. Observability focused on gaining insights into system behaviour through metrics, logs, and traces. As a result, tools like Prometheus, Grafana, and ELK stack (Elasticsearch, Logstash, Kibana) gained popularity.

There was a discussion on Elastic's Github quite a while ago: https://github.com/elastic/kibana/issues/88956 but I haven't found any related documentation on Elastic's website.

ahh good catch here, I have raised a FR to get this added to Kibana https://github.com/elastic/kibana/issues/157348

WriteRequest::timeseries is a vector (https://github.com/prometheus/prometheus/blob/main/prompb/re...) and

Developers widely use Prometheus as a system for operational monitoring and alerting for their projects. Here is a list of tools for monitoring frontend services with Prometheus.

Just to give an example of the power of Go for CLI builds, you may have already used or at least heard of Docker, Kubernetes, Prometheus, Terraform, but what do they all have in common? They all have a large part of their usability via CLI and are developed in Go 🐿.

Distributed system administrators need mechanisms and tools for monitoring individual nodes in order to analyze the system and promptly detect anomalies. Developers also need effective mechanisms for analyzing, diagnosing issues, and identifying bugs in protocol implementations. Logging, tracing, and collecting metrics are common observability techniques to allow monitoring and obtaining diagnostic information from the system; most of the explored code bases use these techniques. OpenTelemetry and Prometheus are popular open-source monitoring solutions, which are used in many of the explored code bases.

Setting up monitoring for a system, especially one involving GRPC communication, provides crucial visibility into its operations. In this guide, we walked through the steps to instrument both a GRPC server and client with Prometheus metrics, exposed those metrics via an HTTP endpoint, and visualized them using Grafana. The Docker-Compose setup simplified the deployment of both Prometheus and Grafana, ensuring a streamlined process.

Alerting and Notification: Select a tool with flexible alerting mechanisms to proactively detect anomalies or deviations from defined thresholds. Consider asking questions like "Does this tool offer customizable alerting options and support notification channels that suit our team's communication preferences?" A tool like Prometheus provides robust alerting capabilities.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit. It is designed to monitor highly dynamic containerized systems, making it an excellent choice for monitoring Docker containers and Kubernetes clusters.

wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz

Prometheus: Open-source monitoring system. Often used together with Grafana.