aws-ebs-csi-driver VS opentelemetry-go

The AWS EBS CSI Driver relies on IAM permissions to communicate with Amazon EBS for volume management on behalf of the user. The example policy can be used to define the required permissions. Additionally, AWS provides a managed policy at ARN arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

I looks like the driver's permissions to invoke the EBS APIs was revoked and/or changed. When you install the EBS CSI addon you can either inherit permissions from the worker node or you can choose an IRSA role (preferred). If you use IRSA, the service account that the EBS CSI driver uses should have an annotation that references the ARN of the IAM role you selected, e.g. eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role. You can see an example of the IAM policy the driver needs here, https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/fb6d456558fb291b13f855454c1525c7acaf7046/docs/example-iam-policy.json.

I'm know it's possible to write terraform code that exhibits that issue, but that's not the case in my experience. I'm using helm to deploy aws's ebs csi driver in the above setup. As you mentioned, if the eks cluster was destroyed before the helm provider attempted to use its API to destroy the helm deployment, it would cause problems. And I don't run into that issue. It's not luck of timing, either - I also have a CI process that deploys all of this, tests, and deletes it all that has succeeded hundreds of times.

Any Kubernetes cluster requires persistent storage - whether organizations choose to begin with an on-premise Kubernetes cluster and migrate to the public cloud, or provision a Kubernetes cluster using a managed service in the cloud. Kubernetes supports multiple types of persistent storage – from object storage (such as Azure Blob storage or Google Cloud Storage), block storage (such as Amazon EBS, Azure Disk, or Google Persistent Disk), or file sharing storage (such as Amazon EFS, Azure Files or Google Cloud Filestore). The fact that each cloud provider has its implementation of persistent storage adds to the complexity of storage management, not to mention a scenario where an organization is provisioning Kubernetes clusters over several cloud providers. To succeed in managing Kubernetes clusters over a long period, knowing which storage type to use for each scenario, requires storage expertise.

Lots of info in this issue: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1163

aws-ebs-csi-driver

Hello, we are running an EKS cluster (1.20) with aws-ebs-csi-driver (1.4.0). After recreating our whole cluster we can observe that the EBS volumes from our PVCs still exist but the "mapping" to the PVCs is gone.

OP could probably just layer their own CSI driver on top of an existing one (a la aws-ebs-csi-driver), but there's still several problems:

Our first approach was to implement a separate SDK for each independent technology stack. We decided to use OpenTelemetry which is widely adopted and covers most of our needs.

Distributed system administrators need mechanisms and tools for monitoring individual nodes in order to analyze the system and promptly detect anomalies. Developers also need effective mechanisms for analyzing, diagnosing issues, and identifying bugs in protocol implementations. Logging, tracing, and collecting metrics are common observability techniques to allow monitoring and obtaining diagnostic information from the system; most of the explored code bases use these techniques. OpenTelemetry and Prometheus are popular open-source monitoring solutions, which are used in many of the explored code bases.

OpenTelemetry

When choosing distributed tracing tools, considerations include your technology stack, business requirements, and monitoring complexity. Zipkin, SkyWalking, and OpenTelemetry are popular distributed tracing solutions, each with its unique features.

You can follow this process with any large token AI system like Claude by identifying tracing data relevant to the code you are working on, using it as context to prompt OpenAI or other LLMs. Generally, you’d generate tracing data by implementing OpenTelemetry (aka OTEL) libraries into your application, adding spans to your functions with Jaeger, or using commercial SaaS tools like Honeycomb and Datadog.

While many programming languages provide robust support for Open Telemetry, this instance focuses on Golang. It's important to note that, in the current context, the logs SDK for Golang is not implemented. For future reference consult the list of supported languages and explore the Open Telemetry repositories. Always prioritize the main repository and its contrib repository, housing extensions and instrumentation libraries crucial to the Open Telemetry framework. Stay updated with the latest developments to ensure seamless integration and enhanced functionality.

OneUptime (https://github.com/oneuptime/oneuptime) is the open-source alternative to DataDog. It's 100% free and you can self-host it on your VM / server / cloud or you can use SaaS at https://oneuptime.com

NEW UPDATES (since we last posted to HN): We now support OpenTelemetry (https://opentelemetry.io/) natively which will help you to monitor, observe and debug any app, service, database or stack.

OpenTelemetry is the fastest growing Cloud Native Computing Foundation (CNCF) project. It standardizes the instrumentation and collection of traces, metrics, and logs from applications, and is supported by all the major observability projects, languages, and tools. One standard to rule them all!

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. https://opentelemetry.io/

Tracetest uses your existing OpenTelemetry traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!