tinc VS vpncloud

But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2].

I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages.

I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss.

Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3].

So I'm currently looking to move the Ceph into WireGuard.

Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break.

[1]: https://github.com/gsliepen/tinc/issues/218

[2]: https://github.com/slackhq/nebula/issues/637

[3]: https://github.com/google/security-research/security/advisor...

From a purely networking perspective, there are far better solutions than tailscale.

Have a look at full mesh VPNs like:

https://github.com/cjdelisle/cjdns

https://github.com/yggdrasil-network/yggdrasil-go

https://github.com/gsliepen/tinc

https://github.com/costela/wesher

These build actual mesh networks where every node is equal and can serve as a router for other nodes to resolve difficult network topologies (where some nodes might not be connected to the internet, but do have connections to other nodes with an internet connection).

Sending data through multiple routers is also possible. They also deal with nodes disappearing and change routes accordingly.

tailscale (and similar solutions like netbird) still use a bunch of "proxy servers" for that. You can set them up on intermediate nodes, but that have to be dealt with manually (and you get two kinds of nodes).

Two other options are Tinc https://tinc-vpn.org/ or Nebula https://www.defined.net/nebula/

And there is Tinc; the OG overlay network. I don't have experience with this. Seemed a bit of a pain to setup. https://tinc-vpn.org

For what its worth I have used the open source Tinc VPN [1] for mesh multihop routing for ages. It is nowhere near as fast as Wireguard but I could envision Tinc incorporating support for Wireguard if the author were so inclined. Like you mentioned Tinc does not mesh with other VPN's AFAIK.

[1] - https://tinc-vpn.org/

This is actually very simple in concept and is just as simple or even simpler to do with tinc (https://tinc-vpn.org).

Since I can use tinc in bridge mode, I can run tinc on the upstream server and on a local machine which then provides access to several physical machines without running extra software on each of those machines, which is particularly useful for machines that are resource limited, like my Macintosh LC II and LC III+:

http://elsie.zia.io/

It'd be nice if it weren't so difficult to get public addresses.

I clicked expected some broken analogy between https://tinc-vpn.org/ and the Catan board game, but instead it is a Catan implementation. Fair enough.

will generate a real-time network graph using the Graphviz DOT language. It's a cool feature that I find quite useful.

[0] https://tinc-vpn.org/

Another tool worth looking at is vpncloud (https://github.com/dswd/vpncloud). I used to use tinc, but switched to vpncloud 2 years ago.

In my use case, I have a modest number of nodes. Although nodes learn of other nodes from each other, I use ansible to keep each node's config updated.

I use vpncloud (and previously, tinc) between docker hosts. So, you have to be careful about interface MTU's inside of docker, particularly if you use containers based on Alpine.

I think one of the reasons is that people confuse physical servers with manual administration. As I said, I do not do manual administration. Nothing ever gets configured on any server by hand. All administration is through ansible.

I only have one ansible setup, and it can work both for virtualized servers and physical ones. No difference. The only difference is that virtualized servers need to be set up with terraform first, and physical ones need to be ordered first and their IPs entered into a configuration file (inventory).

Of course, I am also careful to avoid becoming dependent on many other cloud services. For example, I use VpnCloud (https://github.com/dswd/vpncloud) for communication between the servers. As a side benefit, this also gives me the flexibility to switch to any infrastructure provider at any time.

My main point was that while virtualized offerings do have their uses, there is a (huge) gap between a $10/month hobby VPS and a company with exploding-growth B2C business. Most new businesses actually fall into that gap: you do not expect hockey-stick exponential growth in a profitable B2B SaaS. That's where you should question the usual default choice of "use AWS". I care about my COGS and my margins, so I look at this choice very carefully.

I'm not a fan of IPFS (I've tried it many times, from the beginning), but Hypercore has made significant improvements. If you're looking for FLOSS mutable torrents, that's probably the best we've got right now (as scary as that may be). Cockroachdb, VPNCloud, and raTox might also be worth your time. At the very least, I could see value in making it expensive to play whack-a-mole on those who are willing to host. Ideally, it would be effortless to mirror or contribute back.