tinc VS wg-meshconf

But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2].

I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages.

I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss.

Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3].

So I'm currently looking to move the Ceph into WireGuard.

Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break.

[1]: https://github.com/gsliepen/tinc/issues/218

[2]: https://github.com/slackhq/nebula/issues/637

[3]: https://github.com/google/security-research/security/advisor...

From a purely networking perspective, there are far better solutions than tailscale.

Have a look at full mesh VPNs like:

https://github.com/cjdelisle/cjdns

https://github.com/yggdrasil-network/yggdrasil-go

https://github.com/gsliepen/tinc

https://github.com/costela/wesher

These build actual mesh networks where every node is equal and can serve as a router for other nodes to resolve difficult network topologies (where some nodes might not be connected to the internet, but do have connections to other nodes with an internet connection).

Sending data through multiple routers is also possible. They also deal with nodes disappearing and change routes accordingly.

tailscale (and similar solutions like netbird) still use a bunch of "proxy servers" for that. You can set them up on intermediate nodes, but that have to be dealt with manually (and you get two kinds of nodes).

Two other options are Tinc https://tinc-vpn.org/ or Nebula https://www.defined.net/nebula/

And there is Tinc; the OG overlay network. I don't have experience with this. Seemed a bit of a pain to setup. https://tinc-vpn.org

For what its worth I have used the open source Tinc VPN [1] for mesh multihop routing for ages. It is nowhere near as fast as Wireguard but I could envision Tinc incorporating support for Wireguard if the author were so inclined. Like you mentioned Tinc does not mesh with other VPN's AFAIK.

[1] - https://tinc-vpn.org/

This is actually very simple in concept and is just as simple or even simpler to do with tinc (https://tinc-vpn.org).

Since I can use tinc in bridge mode, I can run tinc on the upstream server and on a local machine which then provides access to several physical machines without running extra software on each of those machines, which is particularly useful for machines that are resource limited, like my Macintosh LC II and LC III+:

http://elsie.zia.io/

It'd be nice if it weren't so difficult to get public addresses.

I clicked expected some broken analogy between https://tinc-vpn.org/ and the Catan board game, but instead it is a Catan implementation. Fair enough.

will generate a real-time network graph using the Graphviz DOT language. It's a cool feature that I find quite useful.

[0] https://tinc-vpn.org/

my experience, i dont know if this is comparable, but from my memory (i have not made any notes on that), i've tried min.io in december and switched to seaweed a weeks ago, because my usecase was transition from local file storage to DFS + also enable our developers to transition from local filesystem to s3. Since my resources are limited (vsphere VM) with 3 hosts + different disks, i tried to set up a 3 vm cluster with minio first, after i did some research on different systems (ceph, longhorn.io, ..) i wanted to have an easy setup-able system, which supports s3. I relied a lot on what people measured and chose min.io first because it supported mount via s3. Then i tried to copy over about 34 million files (mostly few bytes, but can also be 1Gbyte), with a mass of about 4.2TB. I tried different methods, rsync, cp, cp with parallelism,.. and i took me about 3 days to copy over 300GB of data at best. Then i also found out that it was impossible to list files. We have one single folder with over 300k projects (guid) beneath (growing). After that i gave seaweed a shot. Why i did not used it firsthand was documentation was a bit confusing and it did not gave me all the answers i needed as fast as minio did.

Now, my seaweed setup is a 3 vm cluster with 3 disks per vm (1TB) each. I configured a wireguard mesh (https://github.com/k4yt3x/wg-meshconf) between the VMs and configured master and volumes server to talk to each other via wireguard IPs securely. I also configured ufw to only allow communication between http/gRPC ports. I also configured a filer (using leveldb3) to use wireguard IPs (master and volumes) and let it communicate with some specific servers on the outside (ufw).

After that i mounted the filer via weed.mount on that specific server and tried to copy over the same files/folders. after 2 days i copied over about 1.5 TB of the data via rsync. There was also no problem with file listing and accessing the filer from different machines while uploading stuff. But there is a overhead when reading and creating lots of small files. File listing is even faster than local btrfs file listing.

chris is also very nice and fast fixing bugs.

Hey you should post your wg0.conf If you would like to build a WireGuard mesh try this: https://github.com/k4yt3x/wg-meshconf

There are two factors at play here. The client's public IP actually depends on the gateway they use on accessing the internet. You can disable routing and your clients will keep their public IP and general internet access won't go through the VPS. However, if you want the traffic between "clients" also skip the VPS, then you want a mesh network. wesher and wg-meshconf can help you on configuring them.

Looks great!

I've been using wg-meshconf[1] to assist in setting up Wireguard Mesh Networks on Linux for a while, works amazing!

A massive use case is to setup Kubernetes clusters, where network encryption is extremely important.

[1]: https://github.com/k4yt3x/wg-meshconf