nginx-proxy VS caddy-docker

setting up letsencrypt with nginx-proxy and acme-companion

Here is the link, sorry😅

Shit... Here is the link: https://github.com/nginx-proxy/nginx-proxy

I use the nginx-proxy image and it's acme companion image for automated reverse proxy configuration and certbot.

I'm experiencing a weird situation, and am not sure how to go about finding a solution. I am running the nginx-proxy container (https://github.com/nginx-proxy/nginx-proxy) together with the acme-companion container (https://github.com/nginx-proxy/acme-companion) to provide https connections to all my different applications under different subdomains on the same host (currently, for testing purposes: only two other nginx containers with a plain html page).

https://github.com/nginx-proxy/nginx-proxy for reverse proxy

I currently use this (https://github.com/nginx-proxy/nginx-proxy) and I was able to configure anything via env vars. Some minor nginx include files were needed tho. But no UI. So far I was stunned that it works so well. Deploy a new container add 2 env vars. Done. With automated SSL.

We use the nginx-proxy docker image with its acme-companion to have an auto configuring SSL reverse proxy, so it's really easy to deploy images (we do it based on a merge PR into protected release branches).

We use Traefik for Configuration Discovery in Docker Swarm (add a label with the sub-domain to the service and it will automatically be routed to), for simple setup I always found nginx-proxy to be easy to use with containers.

There is also a short section about it on the Docker hub page under the "Adding custom Caddy modules" section: https://hub.docker.com/_/caddy

Caddy uses a number of volumes. Two point directly at files within our project, first our Caddyfile, then our public folder which Caddy will serve live files. The other two are virtual filesystems Docker will create as defined by the master volumes parameters. We can assume the caddy_config volume is where active configuration is stored as it is not discussed on the Caddy Docker Official Image page, so we're copying their parameter exactly, but the caddy_data volume needs some extra discussion. It is used to store a number of things including SSL certificates. By default Docker creates and destroys volumes upon startup and exit. As we want to persist our certificate across sessions we can take advantage of an external Docker volume. These virtual filesystems are created before starting the Docker session for the first time. This can ve done from the command line or more easily from within the Docker Desktop app. Simply choose "Volumes", click the "Create" button and specify caddy_data.

For a reverse proxy I would suggest Caddy. The configuration is super easy and it handles SSL for you out of the box using LetsEncrypt! There’s also an official docker image like PHP and MySQL here https://hub.docker.com/_/caddy

Their image is dead simple to use, the config is fine, and auto-https is baked in. You can use it as a sidecar, proxy for multiple images, or just as a base to add your app into, depending on the complexity of your existing config.

> First, what versions am I getting? Does using `2.5.1-builder` result in a customer built binary that's version `2.5.1`? The command usage [1] of the `xcaddy` command says it falls back to the `CADDY_VERSION` environment variable if it's not set explicitly. Since it's not set explicitly, I go looking for that variable in the Dockerfile [2].

Yeah - the default version it'll build with is the version embedded in the builder image, so in that case, v2.5.1. But really, you can just use always use the latest builder image and specify the version you want in the xcaddy command, i.e. 'xcaddy build v2.5.1 --with ', or any other git ref if not a version (cause we're using Go to build and you can use any git ref, like a commit hash or branch name if you want to try a WIP pull request).

We set it up with a good default so most users wouldn't need to ask that question, it should "just work" for them. But it's a valid question to ask.

> That's some templating language I'm not familiar with and I can't track down where the variable gets set, at least not quickly.

Yeah we're using Gomplate for generating the Dockerfiles for the official Docker Library builds, since we need to make builds for every CPU architecture, and even Windows docker images (I still have no idea why anyone would want those, but alas). Either way, that's an implementation detail of how we automate this stuff, doesn't matter to users.

> Now, what version of `caddy-dns/cloudflare` am I getting?

The latest, if you don't specify a version. The https://github.com/caddy-dns/cloudflare repo doesn't have tagged releases, so it'll just be the latest commit on the master branch. You can specify a specific commit like '--with github.com/caddy-dns/[email protected]' for (as of this writing) the commit just before the latest.

> What other risks come along with building and maintaining my own custom image?

Honestly, none. Maybe problems with plugins not being compatible with eachother, but Caddy's plugin design means that should rarely happen, except if two plugins have the same module ID. But that's up to you to make sure you don't pick two plugins that try to do the same thing.

Because of the way Go builds work, they can always be cross-compiled. We don't use CGO, so builds of Caddy are completely static and have zero dependencies. There's really no risk that it doesn't build in a specific environment, or whatever.

> If I build a custom image, do I let other people I help with the odd tech thing use it or is all the effort for me only? I don't want to become the maintainer of a Docker image others rely on, so I can't even re-use any related config if I help others in the future since they won't have access to the needed image.

Up to you. But that's the exact reason we don't maintain builds with plugins ourselves. There's literally an infinite amount of combinations possible. Some have suggested like "caddy-lite" and "caddy-full" sort of setups where we ship just a few vetted plugins or "yolo give me all the plugins" but that's silly. We don't have the time or resources to vet all the plugins.

From your perspective it might seem like "duh, there should be an official build with Cloudflare", but really it's a pretty small percentage of users who need this.

> Also, a 4 line Docker file looks nice in terms of being simple, but explicitly declaring or even adding comments describing some of the things I pointed out above can save people a lot of time. Even comments with links to the relevant portions of the docs would be super useful.

(as I wrote in my other comment, the docs for this are on https://hub.docker.com/_/caddy)

> The desire for wildcard certificates is to keep things from being discoverable via CTLogs.

It's really trivial for someone to scan until they hit subdomains that return a successful response, if they really cared. This doesn't really protect from anything. Using wildcards for that is a bit of an antipattern.

FYI for Caddy + cloudflare DNS plugin in Docker, you just need to write a Dockerfile like this (see https://hub.docker.com/_/caddy):

    FROM caddy:2.5.1-builder AS builder

I appreciate the reply. I took some time to look at your example so I can give some feedback on where I end up when I think about building / maintaining my own image.

My immediate reaction is that the example is nice as a one-off build, but it's much more complex if I need to set up something I can maintain long term. I might be overthinking it, but in the context of thinking about something I can maintain my thought process is below. The questions are mostly rhetorical.

First, what versions am I getting? Does using `2.5.1-builder` result in a customer built binary that's version `2.5.1`? The command usage [1] of the `xcaddy` command says it falls back to the `CADDY_VERSION` environment variable if it's not set explicitly. Since it's not set explicitly, I go looking for that variable in the Dockerfile [2].

That's some templating language I'm not familiar with and I can't track down where the variable gets set, at least not quickly. I'd probably have to spend an hour learning how those templates work to figure it out. To make a quicker, educated guess, it most likely matches the builder version. The docs said the version can be set to any git ref, so I can explicitly set it to v2.5.1 on the command line [3] to be certain.

Now, what version of `caddy-dns/cloudflare` am I getting? The xcaddy custom builds section of the docs [4] says the version can optionally be specified, but it's not specified in the above example. There aren't any tags in the repo, so it's probably building off `master`. The doc says it functions similar to `go get`, but doesn't explain what the differences are and the default behavior isn't explained either.

The docs for `go get` [6] say it can use a revision, so maybe a specific commit can be used for that, but I'd need to test it since I'm not super familiar with Golang.

What other risks come along with building and maintaining my own custom image? I could end up with a subtly broken build that only occurs in my environment. Portability doesn't guarantee compatibility [7] and building custom images increases the risk of compatibility issues beyond what I get with official images (building and running vs just running). That blog post is a really cool read on it's own BTW.

I need to consider the potential for breakage even if it's miniscule because my Docker infrastructure is self hosted and will be sitting behind my custom built Caddy image. If my custom image breaks, I need a guaranteed way of having access to a previous, known good version. This is as simple as publishing the images externally, but adds an extra step since I'll need an account at a registry and need to integrate pushes to that registry into my build.

If I build a custom image, do I let other people I help with the odd tech thing use it or is all the effort for me only? I don't want to become the maintainer of a Docker image others rely on, so I can't even re-use any related config if I help others in the future since they won't have access to the needed image.

To be fair, I also see things I don't like in the NGINX Proxy Manager Dockerfile [7]. The two that immediately jump out at me are things I consider common mistakes. Both require unlucky timing to fail, but can technically cause failure IMO. The first is using `apt-get update` which will exit 0 on failure and has the potential to leave `apt-get install` running against obsolete versions. The second is using `apt-get update` in multiple parts of a multistage build. If I were doing it I'd run `apt-get update` in a base image and avoid it in the builder + runtime images to guarantee the versions stay the same between the build container and the runtime container.

It took me about 1h to work through that and write this comment, so it's not just a matter of building a Docker image and plugging in the config. There's a lot of nuance that goes into maintaining a Docker image (I'm sure you know that already) and not having an image with the DNS plugin(s) baked in is a show stopper for anyone like me that can't justify maintaining their own.

Also, a 4 line Docker file looks nice in terms of being simple, but explicitly declaring or even adding comments describing some of the things I pointed out above can save people a lot of time. Even comments with links to the relevant portions of the docs would be super useful.

My reason for wanting the Cloudflare DNS plugin is that I have some things I want to run 100% locally without ever exposing them to the internet. The desire for wildcard certificates is to keep things from being discoverable via CTLogs.

I hope that's useful feedback. I realize someone bemoaning the difficulty of running your stuff at home lab / small business scale isn't exactly the target audience in terms of picking up customers that pay the bills. Thanks again for the reply / example.

1. https://github.com/caddyserver/xcaddy#command-usage

2. https://github.com/caddyserver/caddy-docker/blob/master/Dock...

3. https://github.com/caddyserver/caddy/tree/v2.5.1

4. https://github.com/caddyserver/xcaddy#custom-builds

5. https://github.com/caddy-dns/cloudflare/tags

6. https://go.dev/ref/mod#go-get

7. https://www.redhat.com/en/blog/containers-understanding-diff...

8. https://github.com/NginxProxyManager/docker-nginx-full/blob/...

Installing Caddy should be fairly straightforward using the official image - what error messages are you receiving?