kibana VS opentelemetry-dotnet

As you might have guessed, I spend a lot of time thinking about application security - almost every day, in fact. At my day job, I'm constantly pondering how to enhance Kibana's security in a scalable manner without overburdening my already hardworking team. Outside of work, I'm equally dedicated to making Secutils.dev even more valuable to fellow engineers looking for better security tools.

These newsletters are among the best sources to stay up-to-date with the latest happenings in JavaScript, web development, and Node.js. They conveniently categorize content into sections like new releases, articles & tutorials, and code & tools. Since I use JavaScript/TypeScript and Node.js extensively, both in my day job and for Secutils.dev, I have to stay informed about developments in these essential tools. Usually, I quickly scan through the newsletter and focus only on the items that grab my attention — it doesn't consume much time but keeps me well-informed.

This is the second part of my reflection sparked by the recent “2023 State of Open Source Security” report from Snyk. It got me thinking about the price we pay for false positives in software security. In my previous post, “The Cost of False Positives in Software Security, Part 1: Small Applications”, I talked about how true and false positives affect smaller applications like Secutils.dev. Now, I want to take the same idea and apply it to a much larger software that’s a big part of my daily work: Kibana.

False positives in security are something that really bothers me, as I happen to work on security for both large applications like Kibana, with hundreds of contributors, and smaller ones like Secutils.dev, where I'm the sole developer.

The increasing complexity of modern systems led to the rise of AIOps (Artificial Intelligence for IT Operations) and observability practices. AIOps leveraged machine learning algorithms to automate problem detection, analysis, and resolution. Observability focused on gaining insights into system behaviour through metrics, logs, and traces. As a result, tools like Prometheus, Grafana, and ELK stack (Elasticsearch, Logstash, Kibana) gained popularity.

There was a discussion on Elastic's Github quite a while ago: https://github.com/elastic/kibana/issues/88956 but I haven't found any related documentation on Elastic's website.

ahh good catch here, I have raised a FR to get this added to Kibana https://github.com/elastic/kibana/issues/157348

open-telemetry/opentelemetry-dotnet: The OpenTelemetry .NET Client (github.com)

💡Good to know -- If you wish to export traces to Jaeger, you should use the AddJaegerExporter instead of the AddOtlpExporter. Visit the opentelemetry-dotnet repository to see how it's done.

We're going to use OpenTelemetry .NET SDK. Add following nuget dependencies to the project:

We instrument the application with the OpenTelemetry SDK and SqlClient instrumentation library for .NET. First, we add the following NuGet package references to the API’s project file:

It should be handled automatically by OpenTelemetry middlewares. Just look through documentation and samples https://github.com/open-telemetry/opentelemetry-dotnet

Besides that, the propagation fields (traceparent and tracestate) were added in the message header. In the last article, I said that the standard (in the Working Draft (WD) step of the w3c process) recommends to add the propagation fields in the application-properties section by the message publisher. For the current example, I chose to propagate that context in the message header even for AMQP calls as was done in the dotnet OpenTelemetry example. It's important to reinforce that Trace Context: AMQP protocol is not a W3C Recommendation yet. Take a look at the place where the propagation fields were added:

As long as the Azure Functions app knows the instrumentation key from an Application Insights instance, it traces almost everything. OpenTelemetry.NET is one of the Open Telemetry implementations, has recently released v1.0 for tracing. Both metrics and logging are close to GA. However, it doesn't work well with Azure Functions. Therefore, in this post, let's manually implement the tracing at the log level, which is sent to Application Insights.