server-side-tls VS nginx-cluster

Not a document, but there's the Mozilla SSL configurator that's updated with best practices every time they change.

http: ## EXTERNAL ROUTING - Only use if you want to proxy something manually ## routers: ## SERVICES ## services: ## MIDDLEWARES ## middlewares: # Only Allow Local networks local-ipwhitelist: ipWhiteList: sourceRange: - 127.0.0.1/32 # localhost - 192.168.1.1/24 # LAN Subnet # Security headers securityHeaders: headers: customResponseHeaders: X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex" X-Forwarded-Proto: "https" server: "" customRequestHeaders: X-Forwarded-Proto: "https" sslProxyHeaders: X-Forwarded-Proto: "https" referrerPolicy: "same-origin" hostsProxyHeaders: - "X-Forwarded-Host" contentTypeNosniff: true browserXssFilter: true forceSTSHeader: true stsIncludeSubdomains: true stsSeconds: 63072000 stsPreload: true # Only use secure ciphers - https://ssl-config.mozilla.org/#server=traefik&version=2.6.0&config=intermediate&guideline=5.6 tls: options: default: minVersion: VersionTLS12 cipherSuites: - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # generated 2023-04-23, Mozilla Guideline v5.6, HAProxy 2.2.9-2, OpenSSL 1.1.1n, intermediate configuration # https://ssl-config.mozilla.org/#server=haproxy&version=2.2.9-2&config=intermediate&openssl=1.1.1n&guideline=5.6 # intermediate configuration ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets # curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam ssl-dh-param-file /etc/ssl/dhparam2048 defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen stats bind *:9090 stats enable stats uri /stats stats auth 12345678:12345678 stats refresh 30s stats show-legends #----------------------- # Front-end section # ------------------- # frontend fe_mail # receives traffic from clients bind :80 http-response set-header X-Frame-Options SAMEORIGIN http-response set-header X-Content-Type-Options nosniff http-response set-header Strict-Transport-Security max-age=63072000 mode http redirect scheme https code 301 if !{ ssl_fc } bind :443 ssl crt /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem alpn h2,http/1.1 # Exchange Admin Center ACL List acl whitelist src 1.2.3.4/32 acl ecp_req url_beg /ecp http-request deny if ecp_req !whitelist acl xmail hdr(host) -i exchange.external-fqdn.co.uk acl autodiscover url_beg /Autodiscover acl autodiscover url_beg /autodiscover acl mapi url_beg /mapi acl rpc url_beg /rpc acl owa url_beg /owa acl owa url_beg /OWA acl eas url_beg /Microsoft-Server-ActiveSync acl eas url_beg /Microsoft-Server-activeSync acl ecp url_beg /ecp acl ews url_beg /EWS acl ews url_beg /ews acl oab url_beg /OAB acl default_for_mail url_beg / use_backend be_ex2019_owa if xmail owa use_backend be_ex2019_autodiscover if xmail autodiscover use_backend be_ex2019_mapi if xmail mapi use_backend be_ex2019_activesync if xmail eas use_backend be_ex2019_ews if xmail ews use_backend be_ex2019_rpc if xmail rpc use_backend be_ex2019_default if xmail default_for_mail frontend fe_exchange_imaps mode tcp option tcplog bind :993 name imaps default_backend be_exchange_imaps frontend fe_exchange_smtp mode tcp option tcplog bind :25 name smtp default_backend be_exchange_smtp frontend fe_exchange_smtps mode tcp option tcplog bind :587 name smtps default_backend be_exchange_smtps #------------------------------ # Back-end section #------------------------------ backend be_ex2019_autodiscover mode http server mail exchange.internal-fqdn.co.uk:443 check ssl verify none backend be_ex2019_mapi mode http server mail exchange,internal-fqdn.co.uk:443 check ssl verify none backend be_ex2019_rpc mode http server mail exchange.internal-fqdn.co.uk:443 check ssl verify none backend be_ex2019_owa mode http server mail exchange.internal-fqdn.co.uk:443 check ssl verify none backend be_ex2019_activesync mode http server mail exchange.internal-fqdn.co.uk:443 check ssl verify none backend be_exchange_imaps mode tcp server mail exchange.internal-fqdn.co.uk:993 backend be_ex2019_ews mode http server mail exchange.internal-fqdn.co.uk:443 check ssl verify none backend be_ex2019_default mode http server mail exchange.internal-fqdn.co.uk:443 check ssl verify none backend be_exchange_smtp mode tcp server mail exchange.internal-fqdn.co.uk:25 backend be_exchange_smtps mode tcp server mail exchange.internal-fqdn.co.uk:587

Mozilla has a nice tool for this https://ssl-config.mozilla.org/

# specify time period during which the SSL session can be reused ssl_session_timeout 10m; # enable SSL session caching, you can also place this in your default site config. #Note: if you enabled caching in multiple configs nginx will not load. ssl_session_cache shared:SSL:60m; server { listen 80; server_name plex.domain.com; return 301 https://$server_name$request_uri; } #Upstream to Plex upstream plex_backend { #Set this to the IP address that appears in `ifconfig` (NATTED LAN IP or Public IP address) if you want the bandwidth meter in the server status page to work server 127.0.0.1:32400; keepalive 32; } server { send_timeout 100m; #Some players don't reopen a socket and playback stops totally instead of resuming after an extended pause (e.g. Chrome) #Faster resolving, improves stapling time. Timeout and nameservers may need to be adjusted for your location Google's have been used here. resolver 8.8.4.4 8.8.8.8 valid=300s; resolver_timeout 10s; # define default SSL options ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; #Intentionally not hardened for security for player support and encryption video streams has a lot of overhead with something like AES-256-GCM-SHA384. ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:E> #Why this is important: https://blog.cloudflare.com/ocsp-stapling-how-cloudflare-just-made-ssl-30/ ssl_stapling on; ssl_stapling_verify on; #For letsencrypt.org you can get your chain like this: https://esham.io/2016/01/ocsp-stapling ssl_trusted_certificate /etc/letsencrypt/live/plex.domain.com-0001/chain.pem; #Reuse ssl sessions, avoids unnecessary handshakes #Turning this on will increase performance, but at the cost of security. Read below before making a choice. #https://github.com/mozilla/server-side-tls/issues/135 #https://wiki.mozilla.org/Security/Server_Side_TLS#TLS_tickets_.28RFC_5077.29 #ssl_session_tickets on; ssl_session_tickets off; listen 443 ssl http2; listen [::]:443 ssl; server_name plex.domain.com; # These are the paths to your generated Let's Encrypt SSL certificates. ssl_certificate /etc/letsencrypt/live/plex.domain.com-0001/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/plex.domain.com-0001/privkey.pem; # managed by Certbot # To generate your dhparam.pem file, run `openssl dhparam -out /etc/nginx/dhparam.pem 2048` (without the quotes) in your terminal. ssl_dhparam /etc/nginx/dhparam.pem; ssl_ecdh_curve secp384r1; #add_header Content-Security-Policy "default-src 'none'; script-src 'self'; connect-src 'self'; img-src 'self'; style-src 'self'; font-src 'self';"; #Will ensure https is always used by supported browsers which prevents any server-side http > https redirects, as the browser will internally correct any request to https. #Recommended to submit to your domain to https://hstspreload.org as well. #!WARNING! Only enable this if you intend to only serve Plex over https, until this rule expires in your browser it WONT BE POSSIBLE to access Plex via http, remove 'includeSubDomains;' if you only want it to effect your Plex (sub-)domain. #This is disabled by default as it could cause issues with some playback devices it's advisable to test it with a small max-age and only enable if you don't encounter issues. (Haven't encountered any yet) add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always; #Plex has A LOT of javascript, xml and html. This helps a lot, but if it causes playback issues with devices turn it off. (Haven't encountered any yet) gzip on; gzip_vary on; gzip_min_length 1000; gzip_proxied any; gzip_types text/plain text/css text/xml application/xml text/javascript application/x-javascript image/svg+xml; gzip_disable "MSIE [1-6]\."; #Nginx default client_max_body_size is 1MB, which breaks Camera Upload feature from the phones. #Increasing the limit fixes the issue. Anyhow, if 4K videos are expected to be uploaded, the size might need to be increased even more client_max_body_size 0; error_log /var/log/nginx/openmediavault-plex_error.log error; access_log /var/log/nginx/openmediavault-plex_access.log combined; #Forward real ip and host to Plex proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; #When using ngx_http_realip_module change $proxy_add_x_forwarded_for to '$http_x_forwarded_for,$realip_remote_addr' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions; proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key; proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version; #Websockets proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; #Disables compression between Plex and Nginx, required if using sub_filter below. #May also improve loading time by a very marginal amount, as nginx will compress anyway. #proxy_set_header Accept-Encoding ""; #Buffering off send to the client as soon as the data is received from Plex. proxy_redirect off; proxy_buffering off; location / { #Example of using sub_filter to alter what Plex displays, this disables Plex News. #sub_filter ',news,' ','; #sub_filter_once on; #sub_filter_types text/xml; proxy_pass http://plex_backend; } #PlexPy forward example, works the same for other services. #location /plexpy { #proxy_pass http://127.0.0.1:8181; #} }

Mozilla SSL Configuration Generator

This article came at an interesting timing for me, since I recently started to explore building my own CDN node on top of NGINX open source inspired by this article from Fly: https://fly.io/blog/the-5-hour-content-delivery-network/

I've worked with nginx in the past, and didn't have a great experience, so I was apprehensive diving in, but this time was very different. I think njs (their custom JS scripting environment) was a game changer. Support is built in to nginx core, and available by default in their docker containers, so it's much easier to get started with than Lua scripting. Their JS feature support has some quirks (no optional chaining, array destructuring, console.log's don't show up in logs, are some examples of things that threw me off) but overall nothing that blocked me from implementing the functionality I needed, and the integration points within the nginx config felt fairly natural.

I did run into a number of things that were locked behind their commercial offering that made me a bit uncomfortable betting on it for the long term compared to purely open source alternatives. Off the top of my head:

- DNS discovery. There's a thread on the Fly example repo accompanying the blog post that describes the use case and proposes some workarounds: https://github.com/fly-apps/nginx-cluster/issues/2. Life would be a lot simpler if DNS discovery from the commercial offering was just available (i.e. we can outright delete a brittle bash script that makes DNS queries and reloads nginx on a 5 second interval). This was mentioned in the article as something they're planning to open source.

- Access to some kind of shared key-value store for custom caching logic in njs scripts. With Lua we could just connect to Redis, but njs can't seem to establish persistent network connections for now, so that's off the table. This wasn't mentioned in the article, but they did mention in this Github issue that they're planning on open sourcing their keyval module for this use case: https://github.com/nginx/njs/issues/437. I have some use cases where being able to connect to Redis would be ideal, since syncing keyval across a cluster seems to be eventually consistent (https://docs.nginx.com/nginx/admin-guide/high-availability/z...), but for most of my caching use cases it should be sufficient.

So this article, along with their overall willingness to work with the community to identify and bring commercial features into open source (at least from what I've observed across their responses to Github issues) does a lot to alleviate those concerns.

Though at the end of the day, I don't necessarily need every nginx feature to be in open source. I have no problems with paying for great software like nginx to support its development. But as a small bootstrapped founder, their current pricing structure (from what I could gather on the internet is ~ 2k-5k per running instance), is completely prohibitive. It'd probably require a revamp to the way they sell the software (i.e. self-serve onboarding and automatic license provisioning for smaller customers instead of having customers of all sizes go through expensive sales people), but I'd love to see a more progressive pricing structure with a lower barrier to entry for their commercial product.