amazon-eks-pod-identity-webhook VS cloud-provider-aws

Its not specific to EKS, you can find the underlying webhook that injects the "identity" here: https://github.com/aws/amazon-eks-pod-identity-webhook

You have to jump through much of the same hoops you describe, having a public `.well-known` endpoint for example. I have achieved this in the past by putting the OIDC discovery information in an S3 bucket.

You can install the pod identity webhook and AWS cloud provider, csi provider etc on a bare kube cluster and get pretty close to the EKS experience. Not something I’d do for prod, but interesting as a learning exercise.

Yes you absolutely can. https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md

Is your pod running as an unprivileged user? Sounds like https://github.com/aws/amazon-eks-pod-identity-webhook/issues/8 to me.

On EKS, the pod identity webhook is commonly used as the mechanism for adding the necessary parts of the Pod spec. This webhook looks for ServiceAccounts with a specific set of annotations telling it what ARN it can assume and various other settings. When a Pod is created that uses one of these ServiceAccounts, the webhook mutates the Pod using information found in the ServiceAccount annotations.

If you prefer, you could create ServiceAccounts with these details and use the EKS identity webhook, but I don't see kOps supporting that webhook as a native addon.

The Amazon EKS Pod Identity Webhook on the cluster watches for pods that are associated with service accounts with this special annotation & injects Web Identity Token credentials into the pod as environment variables (technical details here).

I don’t know how KOPs manages IAM creds for pods these days, but you can use this (my recommendation) https://github.com/aws/amazon-eks-pod-identity-webhook, or something like KIAM or kube2iam

You can install the pod identity webhook and AWS cloud provider, csi provider etc on a bare kube cluster and get pretty close to the EKS experience. Not something I’d do for prod, but interesting as a learning exercise.

The most naïve way of implementing support for warm pools is to do nothing more than creating the warm pool. Unfortunately, this would start kubelet, which will register the Node with the cluster. Since the AWS cloud provider does not remove instances in stopped state, the control plane marks the Node NotReady, but keep it around in case it comes back up.