Hystrix VS Apache ZooKeeper

Oops, forgot to include the Hystrix link, https://github.com/Netflix/Hystrix

They had to figure out video compression that worked at the volume that they wanted to deliver. They had to build and maintain their own CDN to be able to have a always available and consistent viewing experience. Don’t even get me started on the resiliency tools like hystrix that they were kind enough to open source. I mean, they have their own fucking data science framework and they’re looking into using neural networks to downscale video.. Sound familiar? That’s cause that’s practically the same thing as Nvidia’s DLSS (which upscales instead of downscales).

When breaking up a monolithic app into microservices, the communication between these services becomes vital to the health and performance of the application. Technically, you could incorporate the features to manage this traffic directly into your application. This is what Twitter, Google, and Netflix did with massive internal libraries like Finagle, Stubby, and Hysterix.

I came upon Resilience4J when I was running my talk on the Circuit Breaker pattern. The talk included a demo, and it relied on Hystrix. One day, I wanted to update the demo to the latest Hystrix version and noticed that maintainers had deprecated it in favor of Resilience4J.

Netflix was talking alot about circuit breaks a few years ago, and had the Hystrix project. Looks like Hystrix is discontinued, so I'm not sure if there are good library solutions that are easy to adopt. Overall I don't see it getting talked about that frequently... beyond just exponential backoff inside a retry loop.

- https://github.com/Netflix/Hystrix

Routing, circuit breaking, retries, fail-overs, fault injection — Think of Netflix Ribbon, Hytrix and so on.

Let met explain. Request/Reply can need retry but its latency and retry constraints are far different than Pub/Sub (aka fire and forget). Request/Reply retry patterns are best seen in Netflix Hystrix w/ the bulkhead pattern.

Apache ZooKeeper — a distributed coordination, synchronization, and configuration service (written in Java);

To use Kafka, we also need to deploy a service that keeps configuration informations such as Zookeeper.

Failure Detection and Recovery It’s not enough to have backup systems. It’s also crucial to detect failures quickly. Modern systems employ monitoring tools and rely on distributed coordination systems such as Zookeeper or etcd to identify faults in real-time: once detected, recovery mechanisms are triggered to restore the service.

zookeeper: is (was?) used for secrets management. it was also used as a basic health check, but has been since been replaced.

You can install Kafka from https://kafka.apache.org/quickstart. Because Druid and Kafka both use Apache Zookeeper, I opted to use the Zookeeper deployment that comes with Druid, so didn’t start it with Kafka. Once running, I created two topics for me to post the data into, and for Druid to ingest from:

Please note that we use Zookeeper as the Governance Center.

Next, review deployment complexity such as DB-less versus database-backed deployments. For example, Kong does require running Cassandra or Postgres. Apigee requires Cassandra, Zookeeper, and Postgres to run, while other solutions like Express Gateway and Tyk only require Redis. Apache APISIX uses etcd as its data store, it stores and manages routing-related and plugin-related configurations in etcd in the Data Plane.

ZooKeeper, a system for coordinating distributed nodes, similar to Google's Chubby

ElasticJob achieves distributed coordination through ZooKeeper. In practical scenarios, users may start multiple jobs in the same project simultaneously, all of which use the same Apache Curator client. There are certain risks due to the nature of ZooKeeper and the callback method of Curator in a single event thread.

For now we just need to know that Kafka will be running n brokers acting as a cluster. The Kafka cluster will use Zookeeper to orchestrate the brokers.