haproxy VS Eureka

I wanted to try it out just now but hit a roadblock immediately - it cannot automatically obtain and maintain TLS certificates. You have to use an external client (e.g. acme.sh), set up a cron to check/renew them, and poke HAProxy to reload them if necessary. I'm way past doing this in 2023.

https://www.haproxy.com/blog/haproxy-and-let-s-encrypt

https://github.com/haproxy/haproxy/issues/1864

I turn on 1 relatively cheap cloud server to process DNS, https and stratum connections and route them via haproxy to one of N miner servers described above.

With the recent discussions about memory safe languages, HAProxy is still surprisingly written in C [0].

[0]: https://github.com/haproxy/haproxy

It does not, because HAProxy does not perform any disk access at runtime and thus would be unable to persist the certificates anywhere. Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second.

See this issue and especially the comment from Lukas Tribus: https://github.com/haproxy/haproxy/issues/1864

Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker.

:+1: for the effort to document this, and coordinating the disclosure with the vendors. This mainly talks about rate-limiting bypass/DoS, but if XFF is also used for audit trail logging of IP addresses and/or IP-based access lists, then the security implications can be even more severe, with falsified audit logs and bypassed security controls.

Setting up an application server behind a reverse proxy to use the "real" client IP is unfortunately very typically just a trial-and-error based process, with very little room for this kind of nuanced security-conciousness, because the configuration and exact behavior is all so non-standardized across different implementations of reverse-proxies and application servers... Typically users will just try different configuration settings until they find a combination that seems to work, and you would actually need to dig in with curl and tshark to understand the edge cases, because the documentation of the application-specific implementation is typically just one brief sentence...

Getting XFF working correctly through a complicated HTTP stack with multiple layers of nginx/haproxy/apache proxies (yes, they have different non-overlapping feature sets), custom backends implementing custom XFF handling/forwarding, and jetty/spring backends upgraded across a major version bump that changed the implementation and configuration properties related to XFF handling was insanely difficult. And of course it broke when migrating from a F5 LB to an AWS ALB, because it behaved differently for that one edge-case for an important customer... highly recommended to just override the entire XFF header with a single value at the appropriate point in your stack, if at all possible.

If just the naive leftmost-first vs rightmost-ish-with-configurable-list-of-trusted-upstream-proxies wasn't enough, then yeah, HAProxy does the thing where it adds a new 100% standards-compliant header continuation line [1] that maybe 1% of backend application developers have ever tested with. And trying to configure HAProxy to interpret the incoming XFF headers for logging/access-control ~is~/was even more weird [2].

[1] https://github.com/haproxy/haproxy/issues/44

I was setting up microservices based on Netflix Eureka and experimenting on top of spring-cloud and after weeks of research and development the question rose!

The closest thing I've found is Netflix's Eureka, though its very Java oriented and I found hard to use.

There's an open issue documenting this problem on Spring Cloud Netflix and Netflix Eureka.

Service discovery — Traditionally provided by platforms like Netflix Eureka or Consul.

With multiple services running, it’s hard to discover where they’re located. The dependencies between multiple services are not always easily found, and new services may be deployed with a new dependency on an older service. Those services can be deployed anywhere in the infrastructure, so what you need is a Service Discovery service. There are plenty available, such as Netflix Eureka or HashiCorp Consul.