vault-k8s VS sealed-secrets

enabled: true # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/ image: repository: "hashicorp/vault-k8s" tag: "latest" resources: requests: memory: 256Mi cpu: 250m limits: memory: 256Mi cpu: 250m server: # These Resource Limits are in line with node requirements in the # Vault Reference Architecture for a Small Cluster resources: requests: memory: 8Gi cpu: 2000m limits: memory: 16Gi cpu: 2000m # For HA configuration and because we need to manually init the vault, # we need to define custom readiness/liveness Probe settings readinessProbe: enabled: true path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204" livenessProbe: enabled: true path: "/v1/sys/health?standbyok=true" initialDelaySeconds: 60 # extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be # used to include variables required for auto-unseal. extraEnvironmentVars: VAULT_CACERT: /vault/userconfig/tls-ca/ca.crt # extraVolumes is a list of extra volumes to mount. These will be exposed # to Vault in the path \/vault/userconfig//`. extraVolumes: - type: secret name: tls-ca - type: secret name: tls-listener-1 - type: secret name: tls-server # This configures the Vault Statefulset to create a PVC for audit logs. # See https://www.vaultproject.io/docs/audit/index.html to know more auditStorage: enabled: true storageClass: solidfire-gold dataStorage: enabled: true storageClass: solidfire-gold standalone: enabled: false # Run Vault in "HA" mode. ha: enabled: true replicas: 3 raft: enabled: true setNodeId: true config: |          ui = true          listener "tcp" {            address = "0.0.0.0:8200"            cluster_address = "0.0.0.0:8201"            tls_cert_file = "/vault/userconfig/tls-listener-1/server.crt"            tls_key_file = "/vault/userconfig/tls-listener-1/server.key"            tls_client_ca_file = "/vault/userconfig/tls-ca/ca.crt"          }          storage "raft" {            path = "/vault/data"            retry_join {              leader_api_addr = "https://vault-0.vault-internal:8200"`               leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"               leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"               leader_client_key_file = "/vault/userconfig/tls-server/server.key"             }             retry_join {               leader_api_addr = "https://vault-1.vault-internal:8200"               leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"               leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"               leader_client_key_file = "/vault/userconfig/tls-server/server.key"             }             retry_join {               leader_api_addr = "https://vault-2.vault-internal:8200"               leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"               leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"               leader_client_key_file = "/vault/userconfig/tls-server/server.key"             }             autopilot {               cleanup_dead_servers = "true"               last_contact_threshold = "200ms"               last_contact_failure_threshold = "10m"               max_trailing_logs = 250000               min_quorum = 3               server_stabilization_time = "10s"             }           }           service_registration "kubernetes" {} # Vault UI ui: enabled: true serviceType: "LoadBalancer" externalPort: 8443 loadBalancerIP: 10.193.124.70 loadBalancerSourceRanges: - 0.0.0.0/0

vault: global: enabled: true tlsDisable: false injector: enabled: true # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/ image: repository: "hashicorp/vault-k8s" tag: "latest" resources: requests: memory: 256Mi cpu: 250m limits: memory: 256Mi cpu: 250m server: # These Resource Limits are in line with node requirements in the # Vault Reference Architecture for a Small Cluster resources: requests: memory: 8Gi cpu: 2000m limits: memory: 16Gi cpu: 2000m # For HA configuration and because we need to manually init the vault, # we need to define custom readiness/liveness Probe settings readinessProbe: enabled: true path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204" livenessProbe: enabled: true path: "/v1/sys/health?standbyok=true" initialDelaySeconds: 60 # extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be # used to include variables required for auto-unseal. extraEnvironmentVars: VAULT_CACERT: /vault/userconfig/tls-ca/ca.crt # extraVolumes is a list of extra volumes to mount. These will be exposed # to Vault in the path \/vault/userconfig//`. extraVolumes: - type: secret name: tls-listener - type: secret name: tls-server - type: secret name: tls-ca # This configures the Vault Statefulset to create a PVC for audit logs. # See https://www.vaultproject.io/docs/audit/index.html to know more auditStorage: enabled: true storageClass: solidfire-gold dataStorage: enabled: true storageClass: solidfire-gold standalone: enabled: false # Run Vault in "HA" mode. ha: enabled: true replicas: 3 raft: enabled: true setNodeId: true config: |          ui = true          listener "tcp" {            address = "[::]:8200"            cluster_address = "[::]:8201"            tls_cert_file = "/vault/userconfig/tls-listener/server.crt"            tls_key_file = "/vault/userconfig/tls-listener/server.key"            tls_client_ca_file = "/vault/userconfig/tls-ca/ca.crt"          }          storage "raft" {            path = "/vault/data"            retry_join {              leader_api_addr = "https://vault-0.vault-internal:8200"`               leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"               leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"               leader_client_key_file = "/vault/userconfig/tls-server/server.key"             }             retry_join {               leader_api_addr = "https://vault-1.vault-internal:8200"               leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"               leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"               leader_client_key_file = "/vault/userconfig/tls-server/server.key"             }             retry_join {               leader_api_addr = "https://vault-2.vault-internal:8200"               leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"               leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"               leader_client_key_file = "/vault/userconfig/tls-server/server.key"             }             autopilot {               cleanup_dead_servers = "true"               last_contact_threshold = "200ms"               last_contact_failure_threshold = "10m"               max_trailing_logs = 250000               min_quorum = 5               server_stabilization_time = "10s"             }           }           service_registration "kubernetes" {} # Vault UI ui: enabled: true serviceType: "LoadBalancer" serviceNodePort: null externalPort: 8200

Hashicorp Vault k8s is an operator that modifies pods via a mutating webhook to connect between vault and pod via sidecars (additional containers) to provide secrets. This has the major advantage that no secret objects are created in Kubernetes here. The disadvantage is that this way only works with Vault.

They’ve made it so you can define the order that the vault sidecar starts in, so that the proxy will be running first. https://github.com/hashicorp/vault-k8s/issues/53

External-Secrets Operator : A Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret (Alternatives : Vault, SOPS, Sealed Secrets)

I like sealed secrets (https://github.com/bitnami-labs/sealed-secrets) a lot. It's like 1Password, but for apps in kubernetes. You only need to secure a private key, and can throw encrypted secrets in a public github repo or anywhere you want.

It's owned by VMware (Broadcom) now, so you have to decide which company you hate more.

If you have noticed, you are setting secrets in plain text on the application-configmap.yml file, which is not ideal and is not a best practice for security. The best way to do this securely would be to use AWS Secrets Manager, an external service like HashiCorp Vault, or Sealed Secrets. To learn more about these methods see the blog post Shhhh... Kubernetes Secrets Are Not Really Secret!.

Yeah documentation is hard and I'm guilty (as a former maintainer of SealedSecrets)

SealedSecrets was designed with "write only" secrets in mind.

Turns out a lot of people need to access the current secrets because they need to update a part of a "composite" secret.

There are two kinds of "composite" secrets, one easy and one harder, but if you don't know how to do it, even the easier is hard:

1. Secret with multiple data "items" (also called keys in K8s Secret jargon but that's confusing when there is encryption involved). I.e. good old "data":{"foo": "....", "bar": "..."}

2. Secrets where data within one item is actually a config file with cleartext and secrets mixed up in one single string (usually some JSON or YAML or TOML)

Case 1 is "easy" to deal with once you realize that sealed secrets files are just text files and you can just manually merge and update encryoted data items. We even created a "merge" and some "raw" encryption APIs to make that process a little less "copy pasta" but it's still hard to have a good UX that works for everyone.

Case 2 is harder. We did implement a data templating feature that allows you to generate a config file via a go-template that keeps the cleartext parts in clear and uses templating directives to inject the secret parts where you want (referencing the encrypted the items)

The main problem with case 2 is that it's undocumented.

The feature landed in 2021:

https://github.com/bitnami-labs/sealed-secrets/pull/580

I noticed that people at my current $dayjob used sealed secrets for years and it took me a while to understand that the reason they hated it was that they didn't know about that fundamental feature.

And how to blame them!? It's still undocumented!

In my defense I spent so much effort before and after I left VMware to lobby so that the project got the necessary staffing so it wouldn't die of bitrot that I didn't have much time left to work on documentation. Which is a bit said and probably just an excuse :-)

That said, I'm happy that the project is alive and the current maintainers are taking care of it against the forces of entropy. Perhaps some doc work would be useful too. Unfortunately I don't have time for now.

This might be OT, and forgive me, but I think one of the best practices for Encrypting and Managing secrets in Kubernetes is to use Sealed Secrets, they allow your secrets to be securely stored in git with the rest of the configuration and yet no one with access to the Git repository will be able to read them. I say this might be OT, because Sealed Secrets are trying to mitigate a different threat, the threat of the secrets at rest somewhere, and not "live in the cluster", where in theory all the ingredients to decrypt the secrets would still live.

The addition of Consul and Vault gives me a few things. For one, right now I'm handling secrets with a mixture of SOPS and Sealed Secrets. I use Vault in my professional life, and have used both Vault and Consul at my last job. Vault is a beast, so I may as well get better at it; plus its options for secret injection are better.

Use Sealed Secrets Operator.

sealed-secrets (sealed)