aws-imds-packet-analyzer VS cluster-api-provider-nested

I'm the technical lead for IMDS. Without an IMDS, the alternative is usually hard-coding long-lived credentials, which is much worse.

We've looked a few times at different ways that IMDS could vend different credentials to different user-ids. We documented how to use local firewall rules, which you've linked to. This gives single uid restrictions, similar to filterd. We also have a BPF based tracer tool, https://github.com/aws/aws-imds-packet-analyzer which can monitor which user ids and processes are calling IMDS (and which version they are using).

Our next thought was to expose IMDS as a filesystem. That way ordinary POSIX filesystem permissions could be used to control which user ids could read which credentials, and it would even work on Windows. But our research found that security issues (in applications and libraries that customers run) that grant local file reading privileges are even more common than SSRF issues (in part this because many SSRF issues allow "file://" urls, so they become a subset).

We've looked at interfacing with the kernel keyring and the TPM 2.0 interface (and we now have Nitro TPM) ... but both are difficult to call to user space. The latter doesn't get user ID granularity, and the former is hard to coordinate with from the outside for revoking and rotating credentials.

Kubernetes has a lot of limitations from a multi tenancy perspective.

It's functional, but I think it's not as polished as the rest of Kubernetes which is why Kubernetes has a multi tenancy SIG that spawned the hierarchical namespace controller (https://github.com/kubernetes-sigs/hierarchical-namespaces) and virtual clusters (https://github.com/kubernetes-sigs/cluster-api-provider-nest...)

Virtual Cluster (wg-multitenancy)

Also https://github.com/kubernetes-sigs/cluster-api-provider-nested, similar approach to vcluster, but part of the K8s project.

Approach C This approach provides a way to implement hard isolation among Kubernetes tenants who have no trust between them. This provides segregated master plane components for each tenant by creating a mini virtual cluster on the super Kubernetes cluster. Admins can also create custom resources in those virtual clusters as well. This is provided by projects like VirtualCluster and vCluster.