datree VS jaeger

Hi HN, I’m Shimon, the co-founder of Datree: A policy management solution for Kubernetes. We help DevOps engineers prevent misconfigurations in their Kubernetes by enforcing an organizational policy on their clusters. Engineers can define a custom policy or use one of Datree’s built-in policies, such as NIST/NSA Hardening Guide, EKS Security Best Practices, CIS Benchmark, and more.

Our website is at https://datree.io and our GitHub is here: https://github.com/datreeio/datree

This is not the first time I have shown Datree to the HN community: A little over a year ago, I posted here an earlier version of Datree (https://news.ycombinator.com/item?id=28918850). At that time, Datree consisted of a CLI tool to detect Kubernetes misconfigurations during the development process (locally or in the CI/CD), unlike the version I present today in which the enforcement happens in production.

We built the CLI tool because we detected a big problem among Kubernetes operators: Misconfigurations. Kubernetes is extremely complex and flexible, which makes it very easy to poorly configure it in ways that are not secure. And indeed, we talked to dozens of Kubernetes operators who suffered from various problems, starting with failed audits, all the way to downtime in production, all because of misconfigurations.

Our solution was simple: Give the developers the means to shift-left security testing during the development process with a CLI tool that can be integrated into the CI/CD. We thought this was the best way to approach the problem: It is easiest to fix misconfigurations in the development process before they are deployed to production, it prevents context-switching and relieves resources from the DevOps team.

While the CLI tool was very popular among the open-source community (it got over 6000 stars on GitHub), we soon realized that CI/CD enforcement is not enough. As we talked with Datree’s users, we realized we had made a fundamental mistake: We thought of misconfiguration prevention in technical terms rather than organizational terms.

Indeed, from a technical point of view, it makes sense to shift-left Kubernetes security. But when considering the organizational structure in which it takes place, it simply isn’t enough. DevOps engineers told us that they love the shift-left concept, but they simply cannot rely on the goodwill of the engineers to run a CLI tool locally or to monitor all the pipelines leading to production. They need governance, something to help them stay in control of the state of their clusters.

Moreover, we realized that many companies who use Kubernetes are heavily regulated, and cannot take any chances with their security. Sure, these companies want the engineers to fix misconfigurations during development, but they also want something to make sure that no matter what, their clusters remain misconfiguration-free.

Based on this understanding, we developed a new version of Datree that sits on the cluster itself (rather than in the CI/CD) and protects the production environment by blocking misconfigured resources with an admission webhook. It has a centralized policy management solution to enable governance, and native monitoring to get real-time insights into the state of your Kubernetes.

I look forward to hearing your feedback and answering any questions you may have.

Go is a statically typed, compiled programming language designed at Google, it is syntactically similar to C, but with memory safety, garbage collection, structural typing, and CSP-style concurrency. In my case, I needed to run Go for JSON schema validations, in other cases, you might want to perform a CPU-intensive task or use a CLI tool written in Go.

I worked with both of them and from my experience Gatekeeper is more solid and accountable, I even wrote an article about Gatekeeper. Both Gatekeeper and Kyverno require a lot of heavy lifting work. On the one hand, Gatekeeper will probably require more configuration work however the community and the tool itself are more stable than Kyverno. On the other hand, Kyverno policy-as-code capabilities are much easier to use/understand. This way or another, for me using Kyverno’s policy language or Rego for my policies, wasn’t such a pleasant experience. I personally believe in GitOps and shifting left so if you’re looking for tools I would highly recommend you to review Datree, which is an open-source CLI (Disclaimer: I’m one of the developers at Datree). Datree is a more centralized policy management solution rather than a policy engine. Unlike Kyverno/Gatekeeper Datree was built to help DevOps teams to shift left and practice GitOps by delegating more responsibilities to the developers more efficiently. In practice, Datree already comes with built-in rules and policies along with YAML and schema validation for K8s resources and CRDs such as Argo CRDs. Datree’s policies are written in JSONScheme which is a common solid policy language supported by the community for many years. Additionally, Datree’s CLI also comes with a dashboard app where you can monitor the policies in your organization. You can modify and update your policies, review which policies are being used in practice, and control who can create/delete/update your policies. The major difference is that at the moment, unlike Kyverno/Gatekeeper Datree doesn’t provide native policy enforcement in the Kubernetes cluster at the moment but we expect to release this support very soon. At the moment, we provide a way to scan the cluster using a kubectl plugin. Feel free to check it out :)

$ helm plugin install https://github.com/datreeio/helm-datree Installing helm-datree... https://github.com/datreeio/datree/releases/download/1.0.6/datree-cli_1.0.6_Darwin_x86_64.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 673 100 673 0 0 1439 0 --:--:-- --:--:-- --:--:-- 1469 100 6901k 100 6901k 0 0 1852k 0 0:00:03 0:00:03 --:--:-- 2865k helm-datree is installed. See https://hub.datree.io for help getting started. Installed plugin: datree

GitHub

Jaeger maps the flow of requests and data as they traverse a distributed system. These requests may make calls to multiple services, which may introduce their own delays or errors. https://www.jaegertracing.io/

As engineers at past startups, we often had to debug slow queries, poor load times, inconsistent errors, etc... While tools like Jaegar [2] helped us inspect server-side performance, we had no way to tie user events to the traces we were inspecting. In other words, although we had an idea of what API route was slow, there wasn’t much visibility into the actual bottleneck.

This is where our performance product comes in: we’re rethinking a tracing/performance tool that focuses on bridging the gap between the client and server.

What’s unique about our approach is that we lean heavily into creating traces from the frontend. For example, if you’re using our Next.js SDK, we automatically connect browser HTTP requests with server-side code execution, all from the perspective of a user. We find this much more powerful because you can understand what part of your frontend codebase causes a given trace to occur. There’s an example here [3].

From an instrumentation perspective, we’ve built our SDKs on-top of OTel, so you can create custom spans to expand highlight-created traces in server routes that will transparently roll up into the flame graph you see in our UI. You can also send us raw OTel traces and manually set up the client-server connection if you want. [4] Here’s an example of what a trace looks like with a database integration using our Golang GORM SDK, triggered by a frontend GraphQL query [5] [6].

In terms of how it's built, we continue to rely heavily on ClickHouse as our time-series storage engine. Given that traces require that we also query based on an ID for specific groups of spans (more akin to an OLTP db), we’ve leveraged the power of CH materialized views to make these operations efficient (described here [7]).

To try it out, you can spin up the project with our self hosted docs [8] or use our cloud offering at app.highlight.io. The entire stack runs in docker via a compose file, including an OpenTelemetry collector for data ingestion. You’ll need to point your SDK to export data to it by setting the relevant OTLP endpoint configuration (ie. environment variable OTEL_EXPORTER_OTLP_LOGS_ENDPOINT [9]).

Overall, we’d really appreciate feedback on what we’re building here. We’re also all ears if anyone has opinions on what they’d like to see in a product like this!

[1] https://github.com/highlight/highlight/blob/main/LICENSE

[2] https://www.jaegertracing.io

[3] https://app.highlight.io/1383/sessions/COu90Th4Qc3PVYTXbx9Xe...

[4] https://www.highlight.io/docs/getting-started/native-opentel...

[5] https://static.highlight.io/assets/docs/gorm.png

[6] https://github.com/highlight/highlight/blob/1fc9487a676409f1...

[7] https://highlight.io/blog/clickhouse-materialized-views

[8] https://www.highlight.io/docs/getting-started/self-host/self...

[9] https://opentelemetry.io/docs/concepts/sdk-configuration/otl...

For the request following, something like jeager https://www.jaegertracing.io/, because you are talking more about tracing than necessarily logging. For just monitoring, https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack would be the starting point, then it depends. Nginx gives metrics out of the box, then you can pull in the dashboard like https://grafana.com/grafana/dashboards/14314-kubernetes-nginx-ingress-controller-nextgen-devops-nirvana/ , or full metal with something like service mesh monitoring which would provably fulfil most of the requirements

Have you checked out Jaeger [1]? It is lightweight enough for a personal project, but featureful enough to really help "turn on the lightbulb" with other engineers to show them the difference between logging/monitoring and tracing.

[1] https://www.jaegertracing.io/

From the perspective of the realization of GraphQL infrastructure, the interesting direction is "Finding". How to find the problem? How to find the bottleneck of the system? Distributed Tracing System (DTS) will help answer this question. Distributed tracing is a method of observing requests as they propagate through distributed environments. In our scenario, we have dozens of subgraphs, gateway, and transport layer through which the request goes. We have several tools that can be used to detect the whole lifecycle of the request through the system, e.g. Jaeger, Zipkin or solutions that provided DTS as a part of the solution NewRelic.

Jaeger is an open-source, distributed tracing system that monitors and troubleshoots the flow of requests through complex, microservices-based applications, providing a comprehensive view of system interactions.

However, ensuring fault tolerance in distributed systems is not at all easy. These systems are complex, with multiple nodes or components working together. A failure in one node can cascade across the system if not addressed timely. Moreover, the inherently distributed nature of these systems can make it challenging to pinpoint the exact location and cause of fault - that is why modern systems rely heavily on distributed tracing solutions pioneered by Google Dapper and widely available now in Jaeger and OpenTracing. But still, understanding and implementing fault tolerance becomes not just about addressing the failure but predicting and mitigating potential risks before they escalate.

In this article, we'll use HoneyComb.io as our tracing backend. While there are other tools in the market, some of which can be run on your local machine (e.g., Jaeger), I chose HoneyComb because of their complementary tools that offer improved monitoring of the service and insights into its behavior.

The best way to do this, is with the help of tracing tools such as paid tools such as Honeycomb, or your own instance of the open source Jaeger offering, or perhaps Encore's built in tracing system.

In this example, I will create 3 Node.js services (shipping, notification, and courier) using Amplication, add traces to all services, and show how to analyze trace data using Jaeger.