longhorn VS gopass

I'm looking forward to Longhorn[1] taking advantage of this technology.

[1]: https://github.com/longhorn/longhorn

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use.

The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.

I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.

If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.

I'd love to compare it against Talos https://www.talos.dev/ but their lack of support for a persistent storage partition (only separate storage device) really hurts most small home / office usage I'd want to try.

Hi,i was wondering the same. Found more information here in this document: https://github.com/longhorn/longhorn/blob/v1.5.x/enhancements/20230103-recurring-snapshot-cleanup.md

Like most people on r/homelab, it started out with Plex. Rough timeline/services below:

0. Got a Synology DS413 with 4x WD Red 3TB drives. Use Playstation Media Server to stream videos from it. Eventually find some Busybox stuff to add various functionality to the NAS, but it had a habit of undoing them periodically, which was frustrating. I also experienced my first and (knock on wood) only drive failure during this time, which concluded without fanfare once the faulty drive was replaced, and the array repaired itself.

1. While teaching self Python as an Electrical Distribution Engineer at a utility, I befriended the IT head, who gave me an ancient (I think Nehalem? Quad-core Xeon) Dell T310. Promptly got more drives, totaling 7, and tried various OS / NAS platforms. I had OpenMediaVault for a while, but got tired of the UI fighting me when I knew how to do things in shell, so I switched to Debian (which it's based on anyway). Moved to MergerFS [0] + SnapRAID [1] for storage management, and Plex for media. I was also tinkering with various Linux stuff on it constantly.

1.1 Got tired of my tinkering breaking things and requiring troubleshooting/fixing (in retrospect, this provided excellent learning), so I installed Proxmox, reinstalled Debian, and made a golden image with everything set up as desired so I could easily revert.

1.2 A friend told me about Docker. I promptly moved Plex over to it, and probably around this time also got the *Arr Stack [2] going.

2. Got a Supermicro X9DRi-LN4F+ in a 2U chassis w/ 12x 3.5" bays. Got faster/bigger CPUs (E5-2680v2), more RAM, more drives, etc. Shifted container management to Docker Compose. Modded the BIOS to allow it to boot from a NVMe drive on a PCIe adapter.

2.1 Shifted to ZFS on Debian. Other than DKMS occasionally losing its mind during kernel upgrades, this worked well.

2.2 Forked [3] some [4] Packer/Ansible projects to suit my needs, made a VM for everything. NAS, Dev, Webserver, Docker host, etc. Other than outgrowing (IMO) MergerFS/SnapRAID, honestly at this point I could have easily stopped, and could to this day revert back to this setup. It was dead reliable and worked extremely well. IIRC I was also playing with Terraform at this time.

2.3 Successfully broke into tech (Associate SRE) as a mid-career shift, due largely (according to the hiring manager) to what I had done with my homelab. Hooray for hobbies paying off.

3. Got a single Dell R620. I think the idea was to install either pfSense or VyOS on it, but that never came to fruition. Networking was from a Unifi USG (their tiny router + firewall + switch) and 8-port switch, with some AC Pro APs.

4. Got two more R620s. Kubernetes all the things. Each one runs Proxmox in a 3-node cluster with two VMs - a control plane, and worker.

4.0.1 Perhaps worth noting here that I thoroughly tested my migration plan via spinning up some VMs in, IIRC, Digital Ocean that mimicked my home setup. I successfully ran it twice, which was good enough for me.

4.1 Played with Ceph via Rook, but a. disliked (and still to this day) running storage for everything out of K8s b. kept getting clock skew between nodes. Someone on Reddit mentioned it was my low-power C-state settings, but since that was saving me something like ~50 watts/node, I didn't want to deal with the higher power/heat. I landed on Longhorn [5] for cluster storage (i.e. anything that wasn't being handled by the ZFS pool), which was fine, but slow. SATA SSDs (used Intel enterprise drives with PLP, if you're wondering) over GBe aren't super fast, but they should be able to exceed 30 MBps.

4.1.1 Again, worth noting that I spent literally a week poring over every bit of Ceph documentation I could find, from the Red Hat stuff to random Wikis and blog posts. It's not something you just jump into, IMO, and most of the horror stories I read boiled down to "you didn't follow the recommended practices."

5. Got a newer Supermicro, an X11SSH-F, thinking that it would save power consumption over the older dual-socket I had for the NAS. It turned out to not make a big difference. For some reason I don't recall, I had a second X9DRi-LN4F+ mobo, so I sold the other one with the faster CPUs, bought some cheaper CPUs for the other one, and bought more drives for it. It's now a backup target that boots up daily to ingest ZFS snapshots. I have 100% on-site backups for everything. Important things (i.e. anything that I can't get from a torrent) are also off-site.

6. Got some Samsung PM863 NVMe SSDs mounted on PCIe adapters for the Dells, and set up Ceph, but this time handled by Proxmox. It's dead easy, and for whatever reason isn't troubled by the same clock skew issues as I had previously. Still in the process of shifting cluster storage from Longhorn, but I have been successfully using Ceph block storage as fast (1 GBe, anyway - a 10G switch is on the horizon) storage for databases.

So specifically, you asked what I do with the hardware. What I do, as far as my family is concerned, is block ads and serve media. On a more useful level, I try things out related to my job, most recently database-related (I moved from SRE to DBRE a year ago). I have MySQL and Postgres running, and am constantly playing with them. Can you actually do a live buffer pool resize in MySQL? (yes) Is XFS actually faster than ext4 for large DROP TABLE operations? (yes, but not by much) Is it faster to shut down a MySQL server and roll back to a previous ZFS snapshot than to rollback a big transaction? (often yes, although obviously a full shutdown has its own problems) Does Postgres suffer from the same write performance issue as MySQL with random PKs like UUIDv4, despite not clustering by default? (yes, but not to the same extent - still enough to matter, and you should use UUIDv7 if you absolutely need them)

I legitimately love this stuff. I could quite easily make do without a fancy enclosed rack and multiple servers, but I like them, so I have them. The fact that it tends to help my professional growth out at the same time is a bonus.

[0]: https://github.com/trapexit/mergerfs

[1]: https://www.snapraid.it

[2]: https://wiki.servarr.com

[3]: https://github.com/stephanGarland/packer-proxmox-templates

[4]: https://github.com/stephanGarland/ansible-initial-server

[5]: https://longhorn.io

Been running k3s for personal projects etc for some time now on a cluster of raspberry pies. Why pies? Were cheap at the time and wanted to play with arm. I don’t think I would suggest them right now. Nucs will be much better value for money.

Some notes:

Using helm and helmfile https://github.com/helmfile/helmfile for deployments. Seems to work pretty nicely and is pretty flexible.

As I’m using a consumer internet provider ingress is done through cloudflare tunnels https://github.com/cloudflare/cloudflare-ingress-controller in order to not have to deal with ip changes and not have to expose ports.

Persistent volumes were my main issue when previously attempting this, and what changed everything for me was longhorn. https://longhorn.io Make sure to backup your volumes.

Really hyped for https://docs.computeblade.com/ xD

But, if you are really wanting high availability, then roll a kubernetes cluster, and run clustered storage such as longhorn.io, or rook/ceph.

I was mainly referring to this one https://github.com/longhorn/longhorn/discussions/5931 but yeah I peeked into that one too. I'm not at my computer at the moment, how do I provide a support bundle?

If it's RWX, Longhorn 1.5.0 will support that. https://github.com/longhorn/longhorn/issues/5143

You have now finally deployed an enterprise-grade Kubernetes cluster with k3s. You can now deploy some work on this cluster. Some components to take note of are for ingress, you already have Traefik installed, longhorn will handle storage and Containerd as the container runtime engine.

én gopassolok, de same-same

use a password manager, seriously. I know my setup is overkill, but I've been rocking the yubikey/gopass combo for like 3-4 years now.

Depend on the kind of keys or secrets in general, and the infrastructure you work with. As bare minimum KeePassX/KeePassXC works as personal keys vault (that have a master password), GoPass (+git) as team passwords repository that use GPG keys as encryption, and passphrase for SSH keys. And, of course, trying to be mindful in what I run in my local computer.

Thank you for the details, and pointer to a solution. I've just installed gopass.

I also (in looking through other threads) found https://github.com/gopasspw/gopass and by reading the code learned how TOTP works.

I use gopass, because it is pass compliant and supports multiple recipients / teams which was my initial usecase for it. Just ask if you have any questions about my usage of it!

I'm currently using passwordstore/gopass for password management. It uses my GPG key to encrypt the passwords. The GPG key lives only only my Yubikey. The Yubikey requires a touch for each decryption.

That's true, the simple & fast UI (TUI/GUI) helps a lot. However, I would not extrapolate it to a huge problem. I am person, who have written own pass/passage implementation [0], just because I disliked how many steps I need to make to select the password for the form input, modify it or sync secrets.

Initially, I had used the `gopass`. It is probably the most convenient way to start using the password-store. It is cross-platform, 100% compatible with pass & pass-otp. To copy the password, you basically type the part of the file you are looking for. If you type "gopass show github", it will display a TUI, where you can select the file you are looking for (let's say you have two files "personal/github.com.gpg" and "work/github.com.gpg"). Unfortunately, the search function was far from perfect, and it had a problem with typos like "gtihbu" at the time, when I was using it.

To get rid of this issue, I decided to adapt pass/gopass to use `fzf` [2]. In the same time, my .password-store/ dir was rapidly growing that made me think about implementing pass from scratch. I improved the implementation to have better caching, synchronization between machines/mobile, but more importantly - a simple `secret [arg]` command that will execute `fzf` to list all known creds and simplify selection of the password. Of course, it accepted an argument that was limiting the results, which is great when you need to get back to the previous credential to retype something.

The introduction of `fzf` made it really convenient, and I decided to add more commands with fuzzy search, such as:

- `otp` - limits results files containing TOTP/HOTP token, calculates and copies it to the clipboard.

- `secret-edit`, `secret-remove`, `secret-show`... aliases to sub-commands that open `fzf` command in multi-selection mode, so by utilizing space key I could select what files are meant to be modified, removed, displayed etc. Quite handy for mass-edit.

- `secret-qr` - similar to the gopass' feature, but it made a simplified way to create and display QR codes dedicated to share contacts, WiFI SSID+password combination (etc.) to someone who was asking for creds from me.

Awesome, but alt-tabbing got me annoyed after a few years of using. I started pursuing for more sophisticated interface. I decided to give `rofi` [3] a try. I managed to fork that repo and also adapt to my convention of using password-store, but I left i3 for a macOS.

Currently, I have started working on a browser extension that takes care of suggesting password-store creds (based on the path, input parameters, location on the website etc.) similarly to what uBlock Origin does. That configuration is passed to my pass implementation, so on the github.com, my browser have only "work" and "personal" auto-suggestion, when I am focusing the text input.

I plan to create a similar app to Shortcat [4], but it will preserve the information what password has been asked for the focused app. I think, with VoiceOver assistance, it is more than possible to mitigate the need for alt-tabbing to the terminal for electron/native apps.

[0]: It is a private repository, maybe when it will be polished enough I will open-source it.

[1]: https://github.com/gopasspw/gopass

[2]: https://github.com/junegunn/fzf

[3]: https://github.com/alecdwm/pass-rofi-gui

[4]: https://shortcat.app/

gopass is what I've used for a long time. I like how it interfaces with the yubikey/gpg and how password stores can be held in a git repo. There are browser interfaces and GUIs for it but I tend to use it from the command line most of the time.

gopass :)

I wrote my own secret manager: safe. It stores your secrets as encrypted files on your disk (like pass and gopass), and is accessible from the command line. It differs from them in that you only need a master password to use it (so no GPG keys to manage). It comes with an agent (like ssh-agent) that can store your encryption key in memory to avoid typing your master password over and over.