traefik VS Caddy

K3s comes by default with traefik as the ingress controller. I heard great things about it, but I prefer to use ingress-nginx. This is simply because I'm more familiar with it. You can choose pretty much any ingress controller you want for Kubernetes, so pick one according to your own preferences.

I got some of the way down the rabbithole of implementing Traefik in my homelab before realizing that how I’ve built my lab and some of my services might not be a good fit for Traefik. I started with it on a tip from a coworker, but his use case is a little different than mine - specifically that I already have some robust Ansible roles built for the services I run, and I prefer config management in Ansible than in Docker Compose files.

> As a relatively young dev, the idea of a "web server" as a standalone binary that serves your application (vs a library that you use to write your own "server") feels strange.

In my eyes, the ideal setup is one that's layered: where you have an ingress that's basically a load balancer that also ensures that you have SSL/TLS certificates, enforces rate limits, perhaps is used for some very basic logging, or can optionally do any URL rewriting that you need. Personally, i think that Caddy (https://caddyserver.com/) is lovely for this, whereas some people prefer something like Traefik (https://traefik.io/), though the older software packages like Nginx (https://nginx.org/en/) or even Apache (https://www.apache.org/) are good too, as long as the pattern itself is in place.

Then, you may additionally have any sorts of middleware that you need, such as a service mesh for service discovery, or providing internal SSL/TLS - personally Docker Swarm (https://docs.docker.com/engine/swarm/) overlay networks have always been enough for me in this regard, though some people enjoy other solutions, such as Hashicorp Consul (https://www.consul.io/), or maybe something intended for Kubernetes or other platforms that you already may be using, like Linkerd (https://linkerd.io/).

Finally, you have your actual application with its server. Personally, i think that the web server should be embedded (for example, embedded Tomcat with Spring Boot) or indeed just be a library that's a part of the application executable, as long as you can update it easily enough by rebuilding the application - containers are good for this, but aren't strictly necessary, since sometimes other forms of automation and packaging are also enough.

The reason why i believe this, is because i've seen plenty of deployments where that just isn't the case:

  - attempts to store certificates within the application, each application server having different requirements for the formats to be used, making management (and automation) of renewal a total nightmare

I haven't used this: https://github.com/traefik/traefik You could look at replacing nginx proxy manager with traefik

e.g https://traefik.io/ or https://hub.docker.com/r/jwilder/nginx-proxy

Traefik Labs keeps on doing giant leaps and integrating Consul Connect is another step beyond for Traefik Proxy. This indicates the path of this product is humble but reliable and flexible, with an open-minded philosophy behind that is never scared of comparing and collaborating with other important competitors and actors in the CNCF big landscape picture.

I will assume that you have docker and docker-compose installed. If you need more info on Traefik you can have a look at their website, but I have also written a post on setting up Traefik v2 in detail, but we will touch on that in this post.

Take a look at https://github.com/traefik/traefik/issues/6485 for how to use traefik for redis over swarm clusters.

Run web services behind a proxy. For containers, Traefik is popular. I use the SWAG container created by the linuxserver.io folks. It has NGINX for reverse proxy, Letsencrypt integration to help you get a valid certificate, and has fail2ban integration.

I've started to use a web server called caddy. You should check it out; it's awesome. Configuration is so much simpler, and it has a lot of other advantages also! Built-in lets-encrypt handling -- you don't even have to do anything (other than setup DNS "A" and/or "AAAA" record!)

Have you tried Caddy server? No affiliation just a happy user. It’s open source.

It may or may not be able to replace Nginx depending on your use case. For me Caddy has replaced everything I used Nginx for and more.

https://caddyserver.com/

In case you missed it: https://github.com/caddyserver/caddy/issues/2786

Those issues are a thing of the past. There's no use bringing it up again, tbh.

> like acting as a cache

That's fair, we have a WIP cache module here https://github.com/caddyserver/cache-handler, it should be ready soon!

In my eyes, Caddy is a lovely web server that works pretty well as ingress for container clusters (e.g. Nomad, Docker Swarm etc.). That said, i can't help but to feel that v1 was easier and in some ways nicer to use than v2, even though it's abandoned at this point.

That said, i have certain grievances with most of the web servers out there.

Apache2/httpd - actually decently usable even nowadays, but if the fragmentation of service names (httpd vs apache2, with additional scripts like a2enmod) between different distros doesn't hurt it, then the configuration format and how it does reverse proxying and path rewriting most certainly will. The performance is still passable, no matter what anyone says, my applications still have been the bottleneck in approx. 95% of the cases, though that might change with frameworks like Vert.X or such. The further down you scroll, the less user friendly it becomes: https://httpd.apache.org/docs/2.4/rewrite/remapping.html

Nginx - recently migrated my ingress to it at work, seems pretty okay so far, the configuration format seems to make a bit more sense and probably lies somewhere between Apache and Caddy as far as its ease of use and pleasantness goes. I no longer even need rewrite rules to get websockets working properly, which is nice. And my containers can have all of the necessary config in a single file vs the unnecessary boilerplate fragmentation that httpd forces upon me. For example, both of these seem more passable to me when compared to Apache2: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-... and https://www.nginx.com/blog/creating-nginx-rewrite-rules/

Currently, my biggest gripe is that Nginx kills itself when it cannot resolve an upstream host, for example, while Docker containers are still starting, their health checks haven't passed and therefore their DNS records also haven't been created: https://stackoverflow.com/questions/42720618/docker-nginx-st... The worst part is that none of the suggested answers actually work for me, so i can't have a single Nginx instance be in front of the development environment with about 20 containers, a few of those being down when Nginx is being restarted will not let many of them be used until the startup finishes. Unacceptable.

Caddy - as stated before, i liked v1 more than v2, though the project itself is pretty close to as good as a web server might get. What i don't enjoy is them taking the old docs offline, merely letting you download an archive, nor am i a fan of the current docs, since at the current point in time they are a bit like running "tar --usage": https://caddyserver.com/docs/caddyfile/directives/reverse_pr...

It's nice that there are a few examples for the common use cases, but there probably could be even more, just look at what the PHP documentation has at the bottom for a good example: https://www.php.net/manual/en/function.str-replace.php (crowd sourced, but i like the idea of letting the community contribute useful information like that).

Apart from that, some of the behavior is weird and you will get a 200 when you'd expect to get a 502/404 in most other web servers: https://caddy.community/t/why-does-caddy-return-an-empty-200... which will sometimes be misleading ("Huh, i'm not getting any data in the response to my request, even though the status is 200 in my log, weird...")

Also, i remember when v1 had this "fail-fast" habit of shutting down the entire server when renewing/obtaining a certificate failed, something that i utterly hate when web servers do: https://github.com/caddyserver/caddy/issues/642 Admittedly, things are a bit better now: https://caddyserver.com/docs/automatic-https#errors I just don't understand why web servers can be so opinionated about these things and not provide something like "failure_action" in Docker Compose (https://docs.docker.com/compose/compose-file/compose-file-v3...) so that people can choose between either stopping everything as soon as problems manifest, or continuing with a "best effort" strategy.

If i'm hosting 100 sites behind a reverse proxy, i don't want 99 to be taken down just because 1 of them was misconfigured, the web server should be able to throw out a warning about that one host if i tell it to, and proceed to run the rest 99 as instructed. When no web server forces me to cope with such brittleness will be a good day.

It wasn't a security incident, actually. It's true that "a GitHub bug caused it". It wasn't malicious.

TLDR, a contributor made a tag on their own fork of Caddy, and for some reason our next release used their tag, because it turns out forks in GitHub aren't actual separate repos, but rather "still technically the same repo". It's really strange.

All that happened is that the v2.2.1 release wasn't properly signed with Matt's signing key. There was no problem with the code at all.

We've put in place checks during our CI actions to ensure that releases are always verified to be signed by Matt's key. See https://github.com/caddyserver/caddy/pull/3932

On the Caddy website (https://caddyserver.com/) it says:

> Caddy is both a flexible, efficient static file server and a powerful, scalable reverse proxy.

> LIGHTWEIGHT For all its features, Caddy runs lightly and efficiently with relatively low memory footprint and high throughput.

> STATIC FILES By default, Caddy will serve static files in the current working directory. It's so brilliantly simple and works fast.

> Caddy's HTTP server has a wide array of modern features, high performance, and is easy to deploy.

Emphasis with the italics being mine. All of these seem to hint at performance comparaisons being done in specific domains agaisnt other web servers. For example:

- efficient static file server: I would assume that "efficient" here is compared to other file servers, though that could mean also that it doesn't use much resources

- For all its features, Caddy runs lightly and efficiently with relatively low memory footprint and high throughput: That seem to imply that other servers with all these features are heavier and less efficient? Or that other servers just don't have those features? Relatively low memory footprint and high throughput? Relative to what?

- It's so brilliantly simple and works fast: What does fast mean here? Fast relative to what? Apache? The standard Go static file server? Python SimpleHTTPServer?

- high performance: High compared to what?

My goal here isn't to be confrontational. My point is that if there are all those claims on the main page of Caddy, then you must at some point have measured performance, resource usage and things like that against other servers. I think what people are asking is to substantiate those claims with data. You're right that it's hard to generate data in a way that applies perfectly to everyone use case. But then on what do you base all these claims? Why is Caddy called "the ultimate server"? At least that's why I'm wondering.

Caddy is an awesome web server alternative to nginx and apache (httpd). Caddy is written in Go, is a much more performant and extensible web server (in my opinion). With Caddy, you can host basic files or reverse proxy your API's, which is how I use it. I use Caddy because of its automatic TLS functionality so I don't have to worry about manual creation of certificates and keys.