wg-meshconf VS Nebula

my experience, i dont know if this is comparable, but from my memory (i have not made any notes on that), i've tried min.io in december and switched to seaweed a weeks ago, because my usecase was transition from local file storage to DFS + also enable our developers to transition from local filesystem to s3. Since my resources are limited (vsphere VM) with 3 hosts + different disks, i tried to set up a 3 vm cluster with minio first, after i did some research on different systems (ceph, longhorn.io, ..) i wanted to have an easy setup-able system, which supports s3. I relied a lot on what people measured and chose min.io first because it supported mount via s3. Then i tried to copy over about 34 million files (mostly few bytes, but can also be 1Gbyte), with a mass of about 4.2TB. I tried different methods, rsync, cp, cp with parallelism,.. and i took me about 3 days to copy over 300GB of data at best. Then i also found out that it was impossible to list files. We have one single folder with over 300k projects (guid) beneath (growing). After that i gave seaweed a shot. Why i did not used it firsthand was documentation was a bit confusing and it did not gave me all the answers i needed as fast as minio did.

Now, my seaweed setup is a 3 vm cluster with 3 disks per vm (1TB) each. I configured a wireguard mesh (https://github.com/k4yt3x/wg-meshconf) between the VMs and configured master and volumes server to talk to each other via wireguard IPs securely. I also configured ufw to only allow communication between http/gRPC ports. I also configured a filer (using leveldb3) to use wireguard IPs (master and volumes) and let it communicate with some specific servers on the outside (ufw).

After that i mounted the filer via weed.mount on that specific server and tried to copy over the same files/folders. after 2 days i copied over about 1.5 TB of the data via rsync. There was also no problem with file listing and accessing the filer from different machines while uploading stuff. But there is a overhead when reading and creating lots of small files. File listing is even faster than local btrfs file listing.

chris is also very nice and fast fixing bugs.

Hey you should post your wg0.conf If you would like to build a WireGuard mesh try this: https://github.com/k4yt3x/wg-meshconf

There are two factors at play here. The client's public IP actually depends on the gateway they use on accessing the internet. You can disable routing and your clients will keep their public IP and general internet access won't go through the VPS. However, if you want the traffic between "clients" also skip the VPS, then you want a mesh network. wesher and wg-meshconf can help you on configuring them.

Looks great!

I've been using wg-meshconf[1] to assist in setting up Wireguard Mesh Networks on Linux for a while, works amazing!

A massive use case is to setup Kubernetes clusters, where network encryption is extremely important.

[1]: https://github.com/k4yt3x/wg-meshconf

Nebula - Peer-to-peer overlay network. Developed and used internally by Slack. Similar to Tailscale but completely open source. Doesn't use WireGuard. Written in Go.

(I am a Nebula maintainer.) We recently merged support for gVisor-based services, although it's very new, and I don't know of much experimentation that's been done with it yet: https://github.com/slackhq/nebula/pull/965

Nebula, originally from Slack[0].

Wireguard rightly gets a lot of attention, but Nebula is a really simple and easy to deploy mesh network that is often overlooked.

It does lack a management GUI and that stuff is very much DIY.

[0] https://github.com/slackhq/nebula

Fair enough about the android mobile client... My use case only involves meshing linux appliances across various networks so we only need the nebula core binaries which are under MIT license

https://github.com/slackhq/nebula/blob/master/LICENSE

That's not at all confusing with Slack's Nebula. https://github.com/slackhq/nebula

Sounds like a bunch of your pain points are just related to needing an online CA or ICA. But, looking through the Nebula docs I don't know that it supports things like CRL addresses where you could host the CRL, or OCSP responders. Someone got support for an OCSP responder but never submitted a PR with completed code: https://github.com/slackhq/nebula/issues/72

Nebula is a scalable, cross-platform overlay networking tool focused on performance, simplicity, and security. This portable tool is equally adapted for linking a small number of computers or scaling to connect tens of thousands. It integrates encryption, security groups, certificates, and tunneling into a powerful, cohesive connectivity solution. Thanks for the recommendation go to jmeador42.

Replying to my own comment as I can no longer edit it:

The folks over at Slack had an interesting discussion regarding the the "battle of the VPNs" article published by Netmaker I sourced in my parent comment:

https://github.com/slackhq/nebula/discussions/911

Interesting. I thought recognized the logo, apparently seems to be a commercial support offering of https://github.com/slackhq/nebula and they support the "nebula" iOS app. I had been using for nebula/defined in the past.

Nebula: Is super easy to get running. It uses an interesting angle, working on the service and not just the device level. Unfortunately their NAT support seems to be still quite problematic and I am not going to maintain all those forwarded ports manually. There is a PR to support PCP but even if that ever gets applied I am not sure how well that will play with older routers. While it should be battle proven at slack, the community seems to be not that active. It still has the in-house tool that just got released.