sealed-secrets VS HomeLab

If you have noticed, you are setting secrets in plain text on the application-configmap.yml file, which is not ideal and is not a best practice for security. The best way to do this securely would be to use AWS Secrets Manager, an external service like HashiCorp Vault, or Sealed Secrets. To learn more about these methods see the blog post Shhhh... Kubernetes Secrets Are Not Really Secret!.

Yeah documentation is hard and I'm guilty (as a former maintainer of SealedSecrets)

SealedSecrets was designed with "write only" secrets in mind.

Turns out a lot of people need to access the current secrets because they need to update a part of a "composite" secret.

There are two kinds of "composite" secrets, one easy and one harder, but if you don't know how to do it, even the easier is hard:

1. Secret with multiple data "items" (also called keys in K8s Secret jargon but that's confusing when there is encryption involved). I.e. good old "data":{"foo": "....", "bar": "..."}

2. Secrets where data within one item is actually a config file with cleartext and secrets mixed up in one single string (usually some JSON or YAML or TOML)

Case 1 is "easy" to deal with once you realize that sealed secrets files are just text files and you can just manually merge and update encryoted data items. We even created a "merge" and some "raw" encryption APIs to make that process a little less "copy pasta" but it's still hard to have a good UX that works for everyone.

Case 2 is harder. We did implement a data templating feature that allows you to generate a config file via a go-template that keeps the cleartext parts in clear and uses templating directives to inject the secret parts where you want (referencing the encrypted the items)

The main problem with case 2 is that it's undocumented.

The feature landed in 2021:

https://github.com/bitnami-labs/sealed-secrets/pull/580

I noticed that people at my current $dayjob used sealed secrets for years and it took me a while to understand that the reason they hated it was that they didn't know about that fundamental feature.

And how to blame them!? It's still undocumented!

In my defense I spent so much effort before and after I left VMware to lobby so that the project got the necessary staffing so it wouldn't die of bitrot that I didn't have much time left to work on documentation. Which is a bit said and probably just an excuse :-)

That said, I'm happy that the project is alive and the current maintainers are taking care of it against the forces of entropy. Perhaps some doc work would be useful too. Unfortunately I don't have time for now.

This might be OT, and forgive me, but I think one of the best practices for Encrypting and Managing secrets in Kubernetes is to use Sealed Secrets, they allow your secrets to be securely stored in git with the rest of the configuration and yet no one with access to the Git repository will be able to read them. I say this might be OT, because Sealed Secrets are trying to mitigate a different threat, the threat of the secrets at rest somewhere, and not "live in the cluster", where in theory all the ingredients to decrypt the secrets would still live.

The addition of Consul and Vault gives me a few things. For one, right now I'm handling secrets with a mixture of SOPS and Sealed Secrets. I use Vault in my professional life, and have used both Vault and Consul at my last job. Vault is a beast, so I may as well get better at it; plus its options for secret injection are better.

Use Sealed Secrets Operator.

sealed-secrets (sealed)

An option that easily works with GitOps is the Operator Sealed Secrets from Bitnami. Secrets encrypted with it can only be decrypted by operators running inside the cluster, not even by the original author. For encryption, there is a CLI (and a third-party web UI) that requires a connection to the cluster. The disadvantage of this is that the key material is stored in the cluster, the secrets are bound to the cluster and one has to take care of backups and operation.

And if you are interested how I've done it: https://github.com/Michaelpalacce/HomeLab

Example of a service that is being backed up (see backup.velero.io/backup-volume): https://github.com/Michaelpalacce/HomeLab/blob/master/Helm/apps/uptimekuma/templates/deployment.yaml

But for me https://github.com/Michaelpalacce/HomeLab/tree/master/cluster/homelab/apps

So I've been trying to get botkube to work, but it seems to me like the helm chart documentation is outdated? I've been looking at the values and trying to make things work and it didn't really work for me.. https://github.com/Michaelpalacce/HomeLab/blob/master/cluster/homelab/apps/botkube/helmrelease.yaml is what I tried

And here is some documentation and specifics I've ran into. https://github.com/Michaelpalacce/HomeLab/blob/master/docs/Backups.md

Also for configuration storage and git, I recommend taking a gitops approach to things. You can checkup FluxCD2. Here is my repo for reference https://github.com/Michaelpalacce/HomeLab

A few suggestions and thinks I wish I knew when I started ( personal opinions incoming ): 1. Get a single well sized server ( I'd go for 8 cpus .. And like more than 16 gigs of ram and 250+ ssd, ideally m.2 nvme ). Don't bother with raspberry pis.. Check my profile for my previous post to see a comparison of raspberry pi cluster and a x86 cluster... 2. Install proxmox to do virtualization 3. The majority of People in this sub hate kubernetes and the minority is scared to speak, but install k3s on the vms you spin up, this way you can scale resources and not worry about some elaborate setup of which containers go to which node and how to setup tls offloading, load balancing and much much much more as kubernetes will handle it for you. ( bonus since you are a devops engineer so you are probably used to kube clusters ) 4. Use Velero to do restic backups to s3 for services that are critical like dbs and password managers etc. 5. Do everything with ansible or GitOps! My personal homelab is here: https://github.com/Michaelpalacce/HomeLab utilizing ansible, flux for GitOps and renovate to keep my services up to date ( note to say my cluster spin up isn't as nice as I want it to be since when provisioning I have a Dependency on some secrets which I do manually.. But this is only first cluster setup ) 6. Setup a VPN and administrative services on the raspberry pi you have lying around. Personally my VPN is on the router but if yours doesn't support it, do that. 7. After you are done setting everything up and you have a backup, format the drives and reinstall the vms and re set everything up. Document the process, write automation scripts until satisfied... I genuinely recommend you do this like 3 times...... 8. Vpn I guess a lot of people like wireguard or adguard home... Doesn't really matter as long as it's not open vpn and it supports multiple threads. 9. Setup nodered for some automation around your new home, you'll love it. 10. Firefly is amazing! 11. Make sure that your CSI ( Container storage interface ) supports replication. Keep 3 replicas of critical volumes! 11. If you need any help, don't forget to ask :) personally shoot me a dm and I'll do my best to help, but I'm sure most of us here would