hierarchical-namespaces VS aws-imds-packet-analyzer

HNC_VERSION=v1.1.0 HNC_VARIANT=default kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/latest/download/hnc-manager.yaml

Kubernetes has a lot of limitations from a multi tenancy perspective.

It's functional, but I think it's not as polished as the rest of Kubernetes which is why Kubernetes has a multi tenancy SIG that spawned the hierarchical namespace controller (https://github.com/kubernetes-sigs/hierarchical-namespaces) and virtual clusters (https://github.com/kubernetes-sigs/cluster-api-provider-nest...)

Kyverno's a great option. Depending on the usecase you might want to consider https://github.com/kubernetes-sigs/hierarchical-namespaces as well (disclaimer: I'm the original author) - it's good if groups of related namespaces need related objects.

❯ HNC_VERSION=v1.0.0 ❯ kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/${HNC_VERSION}/default.yaml namespace/hnc-system created customresourcedefinition.apiextensions.k8s.io/hierarchyconfigurations.hnc.x-k8s.io created customresourcedefinition.apiextensions.k8s.io/hncconfigurations.hnc.x-k8s.io created customresourcedefinition.apiextensions.k8s.io/subnamespaceanchors.hnc.x-k8s.io created role.rbac.authorization.k8s.io/hnc-leader-election-role created clusterrole.rbac.authorization.k8s.io/hnc-admin-role created clusterrole.rbac.authorization.k8s.io/hnc-manager-role created clusterrole.rbac.authorization.k8s.io/hnc-proxy-role created rolebinding.rbac.authorization.k8s.io/hnc-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/hnc-manager-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/hnc-proxy-rolebinding created secret/hnc-webhook-server-cert created service/hnc-controller-manager-metrics-service created service/hnc-webhook-service created deployment.apps/hnc-controller-manager created mutatingwebhookconfiguration.admissionregistration.k8s.io/hnc-mutating-webhook-configuration created validatingwebhookconfiguration.admissionregistration.k8s.io/hnc-validating-webhook-configuration created # Install helper plugin ❯ kubectl krew install hns

I would say it’s an anti-pattern since using a namespace for multiple environments will be a pain. Not sure what you mean by CRDs though. There is an addon that gives you namespace hierarchies. I.e. each team gets a namespace and they can have sub-namespaces for environments. Check it out: https://github.com/kubernetes-sigs/hierarchical-namespaces

Looking into the Hierarchical Namespace Controller to see if it can simplify our heavily multi-tenanted clusters. So far so good!

HNC is designed for these kinds of scenarios: https://github.com/kubernetes-sigs/hierarchical-namespaces

Project HNC

I'm the technical lead for IMDS. Without an IMDS, the alternative is usually hard-coding long-lived credentials, which is much worse.

We've looked a few times at different ways that IMDS could vend different credentials to different user-ids. We documented how to use local firewall rules, which you've linked to. This gives single uid restrictions, similar to filterd. We also have a BPF based tracer tool, https://github.com/aws/aws-imds-packet-analyzer which can monitor which user ids and processes are calling IMDS (and which version they are using).

Our next thought was to expose IMDS as a filesystem. That way ordinary POSIX filesystem permissions could be used to control which user ids could read which credentials, and it would even work on Windows. But our research found that security issues (in applications and libraries that customers run) that grant local file reading privileges are even more common than SSRF issues (in part this because many SSRF issues allow "file://" urls, so they become a subset).

We've looked at interfacing with the kernel keyring and the TPM 2.0 interface (and we now have Nitro TPM) ... but both are difficult to call to user space. The latter doesn't get user ID granularity, and the former is hard to coordinate with from the outside for revoking and rotating credentials.