nginx-cluster VS caddy-l4

This article came at an interesting timing for me, since I recently started to explore building my own CDN node on top of NGINX open source inspired by this article from Fly: https://fly.io/blog/the-5-hour-content-delivery-network/

I've worked with nginx in the past, and didn't have a great experience, so I was apprehensive diving in, but this time was very different. I think njs (their custom JS scripting environment) was a game changer. Support is built in to nginx core, and available by default in their docker containers, so it's much easier to get started with than Lua scripting. Their JS feature support has some quirks (no optional chaining, array destructuring, console.log's don't show up in logs, are some examples of things that threw me off) but overall nothing that blocked me from implementing the functionality I needed, and the integration points within the nginx config felt fairly natural.

I did run into a number of things that were locked behind their commercial offering that made me a bit uncomfortable betting on it for the long term compared to purely open source alternatives. Off the top of my head:

- DNS discovery. There's a thread on the Fly example repo accompanying the blog post that describes the use case and proposes some workarounds: https://github.com/fly-apps/nginx-cluster/issues/2. Life would be a lot simpler if DNS discovery from the commercial offering was just available (i.e. we can outright delete a brittle bash script that makes DNS queries and reloads nginx on a 5 second interval). This was mentioned in the article as something they're planning to open source.

- Access to some kind of shared key-value store for custom caching logic in njs scripts. With Lua we could just connect to Redis, but njs can't seem to establish persistent network connections for now, so that's off the table. This wasn't mentioned in the article, but they did mention in this Github issue that they're planning on open sourcing their keyval module for this use case: https://github.com/nginx/njs/issues/437. I have some use cases where being able to connect to Redis would be ideal, since syncing keyval across a cluster seems to be eventually consistent (https://docs.nginx.com/nginx/admin-guide/high-availability/z...), but for most of my caching use cases it should be sufficient.

So this article, along with their overall willingness to work with the community to identify and bring commercial features into open source (at least from what I've observed across their responses to Github issues) does a lot to alleviate those concerns.

Though at the end of the day, I don't necessarily need every nginx feature to be in open source. I have no problems with paying for great software like nginx to support its development. But as a small bootstrapped founder, their current pricing structure (from what I could gather on the internet is ~ 2k-5k per running instance), is completely prohibitive. It'd probably require a revamp to the way they sell the software (i.e. self-serve onboarding and automatic license provisioning for smaller customers instead of having customers of all sizes go through expensive sales people), but I'd love to see a more progressive pricing structure with a lower barrier to entry for their commercial product.

https://github.com/mholt/caddy-l4 and https://github.com/kadeessh/kadeessh can do SSH forwarding.

3) Use a L4 TCP/UDP plugin for caddy. https://github.com/mholt/caddy-l4

Wireguard gives my service servers their own internal IP for the gateway to reference (nothing fancy done with it, no iptables modifications like you may see on other guides), and I use NGINX for the game server proxying, specifically linuxserver's nginx container. I love Caddy, but even with caddy-l4 I couldn't get it working right for Valheim (and thus UDP), but NGINX worked real quick.

This may help: https://github.com/mholt/caddy-l4

Well, that's a bit off-topic from the parent comment, which was more about the Caddyfile supporting complex config (versus the underlying JSON config) and not really "complex usecases".

But that said, from a quick Google search... was this an RTMP stream? If so, I suppose you'd want to use https://github.com/mholt/caddy-l4 which is a plugin for Caddy that lets you do TCP-layer things. Caddy's standard distribution just ships an HTTP server (plus TLS and PKI, etc), which is layer-7

You might be able to use caddy-l4's "tee" handler to pipe into multiple "proxy" handlers. But I'm not sure anyone's tried this yet, I had no idea people did this sort of thing. I'd be interested to hear if it does work though.

If you are only having your services accessible via LAN, HTTPS isn't totally necessary, but I would still recommend it. I think a reverse proxy will be easier than your described method. Just set it to listen to 443 and have all of your other services on random ports being proxied from the reverse proxy. If you want HTTPS from your reverse proxy to your services, most reverse proxies will have this kind of feature. Here is the caddy L4 raw TCP stream module: https://github.com/mholt/caddy-l4

I had a similar problem a while back and found this project (Caddy-L4). It had no releases or examples on how to build it so I forked it and added some Docker stuff.

"Caddy L4" aka "Project Conncept" might be what you're looking for:

https://github.com/mholt/caddy-l4

"Project Conncept is an experimental layer 4 app for Caddy. It facilitates composable handling of raw TCP/UDP connections based on properties of the connection or the beginning of the stream."