burrito VS Nebula

Most of the BEAM isn't well-suited for trends in today's immutable architecture world (Docker deploys on something like Kubernetes or ECS). Bootup time on the VM can be long compared to running a Go or OCaml binary, or some Python applications (I find larger Python apps tend to spend a ton of time loading modules). Compile times aren't as fast as Go, so if a fresh deploy requires downloading modules and compile-from-scratch, that'll be longer than other stacks. Now, if you use stateful deploys and hot-code reloading, it's not so bad, but incorporating that involves a bit more risk and specific expertise that most companies don't want to roll into. Basically, the opposite of this article https://ferd.ca/a-pipeline-made-of-airbags.html

Macros are neat but they can really mess up your compile times, and they don't compose well (e.g. ExConstructor and typed_struct and Ecto Schemas all operate on Elixir Structs, but you can't use all three)

If your problem is CPU-bound, there are much better choices: C++, Rust, C. Python has a million libraries that use great FFI so you'll be fine using that too. Ditto memory-bound: there are better languages for this.

This is also not borne from direct experience, but: my understanding is the JVM has a lot more knobs to tune GC. The BEAM GC is IMO amazing, and did the right thing from the beginning to prevent stop-the-world pauses, but if you care about other metrics (good list in this article https://blog.plan99.net/modern-garbage-collection-911ef4f8bd...) you're probably better off with a JVM language.

While the BEAM is great at distribution, "distributed Erlang" (using the VM's features instead of what most companies do, and ad-hoc it with containers and infra) makes assumptions that you can't break, like default k-clustering (one node must be connected to all other nodes). This means you can distribute to some number of nodes, but it's hard to use Distributed Erlang for hundreds or thousands of nodes.

Deployment can be mixed, depending on what you want. BEAM Releases are nice but the lack some of the niceness of direct binaries. Libraries can work around this (like Burrito https://github.com/burrito-elixir/burrito).

If you like static types, Dialyzer is the worst of the "bolted-on" type checkers. mypy/pyright/pyre, Sorbet, Typescript are all way better, since Dialyzer only does "success typing," and gives way worse messages.

   [1]: https://morepablo.com/2023/05/where-have-all-the-hackers-gone.html

The answer was given by the Elixir community with burrito which enables users to pack up everything an Elixir application needs within a binary namely Zig Archiver to package the binary and Zig Wrapper that wraps the Erlang Virtual Machine to be used in multiple platforms (Zig + Rust in the same project 🤯).

Might be worth looking into burrito for that use case?

Elixir is extremely practical for building systems, I know some sysadmin/devops that write their tools in it - which is maybe a bit of a leap for most. It has better support for cli stuff these days but it's not it's strong suit - you can create single-bin packages with stuff like https://github.com/burrito-elixir/burrito or regular "mix releases". (LiveView is very sexy.) It's not statically typed. There is some experimental skunkworks project to add typing to it but probably wont see any public preview until mid/late next year as I understand it.

I would consider 1 to be the major blocker but Burrito has addressed many of the concerns here, including cross-compilation. The only downside of Burrito is that the first boot has to unpack the runtime (which is sub-second in my experience).

Sure! Elixir's been great. Phoenix is a joy to work with, and many of the concurrency primitives built into OTP make it the perfect foundation for a product like this. And rustler makes it super easy to add low-level / native code.

I will say the big downside to using Elixir is that distributing releases is a bit cumbersome. `mix release` expects that you're building on the same OS / version as you'll be running on, though we're looking into using something like burrito [1] aim to alleviate this.

[1] https://github.com/burrito-elixir/burrito

I was intrigued so I went to hunt for the Burrito repo [1].

I thought it was some sort of Erlang native compiler written in Zig (which sounds like an incredible pain in the ass), but it's really "just" a cross-platform installer. Still useful !

[1]: https://github.com/burrito-elixir/burrito/issues?q=is%3Aissu...

(I am a Nebula maintainer.) We recently merged support for gVisor-based services, although it's very new, and I don't know of much experimentation that's been done with it yet: https://github.com/slackhq/nebula/pull/965

Nebula, originally from Slack[0].

Wireguard rightly gets a lot of attention, but Nebula is a really simple and easy to deploy mesh network that is often overlooked.

It does lack a management GUI and that stuff is very much DIY.

[0] https://github.com/slackhq/nebula

Fair enough about the android mobile client... My use case only involves meshing linux appliances across various networks so we only need the nebula core binaries which are under MIT license

https://github.com/slackhq/nebula/blob/master/LICENSE

That's not at all confusing with Slack's Nebula. https://github.com/slackhq/nebula

Sounds like a bunch of your pain points are just related to needing an online CA or ICA. But, looking through the Nebula docs I don't know that it supports things like CRL addresses where you could host the CRL, or OCSP responders. Someone got support for an OCSP responder but never submitted a PR with completed code: https://github.com/slackhq/nebula/issues/72

Nebula is a scalable, cross-platform overlay networking tool focused on performance, simplicity, and security. This portable tool is equally adapted for linking a small number of computers or scaling to connect tens of thousands. It integrates encryption, security groups, certificates, and tunneling into a powerful, cohesive connectivity solution. Thanks for the recommendation go to jmeador42.

Replying to my own comment as I can no longer edit it:

The folks over at Slack had an interesting discussion regarding the the "battle of the VPNs" article published by Netmaker I sourced in my parent comment:

https://github.com/slackhq/nebula/discussions/911

Interesting. I thought recognized the logo, apparently seems to be a commercial support offering of https://github.com/slackhq/nebula and they support the "nebula" iOS app. I had been using for nebula/defined in the past.

Nebula: Is super easy to get running. It uses an interesting angle, working on the service and not just the device level. Unfortunately their NAT support seems to be still quite problematic and I am not going to maintain all those forwarded ports manually. There is a PR to support PCP but even if that ever gets applied I am not sure how well that will play with older routers. While it should be battle proven at slack, the community seems to be not that active. It still has the in-house tool that just got released.

The catch is that I want this to be reliable and fault tolerant, so if some of the game servers in the network go down, the remaining online servers should still always be able to receive broadcasts from any other online server. The servers can also be in multiple geographic locations and I am planning on using a mesh overlay network like Nebula to connect them. Essentially each pair of online servers will likely have a secure link between them that goes directly through the underlying network.