etcd VS Apache ZooKeeper

> https://github.com/etcd-io/etcd/issues/9771

> stale bot marked this as completed (by fucking closing it)

Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly

Using a service discovery mechanism: A service discovery mechanism, such as etcd or ZooKeeper, can help to manage the complexity of microservices by providing a centralized registry of available services and their instances.

etcd is an open-source leader-based distributed key-value datastore designed by a vibrant team of engineers at CoreOS in 2013 and donated to Cloud Native Computing Foundation (CNCF) in 2018. Since then, etcd has grown to be adopted as a datastore in major projects like Kubernetes, CoreDNS, OpenStack, and other relevant tools. etcd is built to be simple, secure, reliable, and fast (benchmarked 10,000 writes/sec), it is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log. etcd is strongly consistent because it has strict serializability, which means a consistent global ordering of events, to be practical, no client subscribed to an etcd database will ever see a stale database (this isn't the case for NoSQl databases the eventual consistency of NoSQL databases ). Also unlike traditional SQL databases, etcd is distributed in nature, allowing high availability without sacrificing consistency.

ETCD_VERSION='3.5.4' wget https://github.com/etcd-io/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz tar -xvf etcd-v${ETCD_VERSION}-linux-amd64.tar.gz && cd etcd-v${ETCD_VERSION}-linux-amd64 cp -a etcd etcdctl /usr/bin/ nohup etcd >/tmp/etcd.log 2>&1 & etcd

ETCD_VER=v3.3.15 # choose either URL GOOGLE_URL=https://storage.googleapis.com/etcd GITHUB_URL=https://github.com/etcd-io/etcd/releases/download DOWNLOAD_URL=${GOOGLE_URL} rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz rm -rf /usr/local/etcd && mkdir -p /usr/local/etcd curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/etcd --strip-components=1 rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz /usr/local/etcd/etcd --version ETCDCTL_API=3 /usr/local/etcd/etcdctl version # start etcd server /usr/local/etcd/etcd -name etcd-1 -listen-peer-urls http://10.0.1.1:2380 -listen-client-urls http://10.0.1.1:2379,http://127.0.0.1:2379 -advertise-client-urls http://10.0.1.1:2379,http://127.0.0.1:2379

Etcd is a key value store for all cluster data. It is an etcd data store. So, It is highly available, reliable, and distributed.

Look at the code for the watcher client[1] and lease management[2].

[1]: https://github.com/etcd-io/etcd/blob/main/client/v3/watch.go

[2]: https://github.com/etcd-io/etcd/blob/main/client/v3/lease.go

Did I miss it or is there no call to os.File.Sync() anywhere?

Since you mention etcd/wal:

https://github.com/etcd-io/etcd/blob/v3.3.27/wal/wal.go#L671

https://github.com/etcd-io/etcd/blob/v3.3.27/pkg/fileutil/sy...

Using a service discovery mechanism: A service discovery mechanism, such as etcd or ZooKeeper, can help to manage the complexity of microservices by providing a centralized registry of available services and their instances.

Apache ZooKeeper: A centralized service to generate unique IDs and prevent race conditions between instances.

Apache ZooKeeper — a distributed coordination, synchronization, and configuration service (written in Java);

To use Kafka, we also need to deploy a service that keeps configuration informations such as Zookeeper.

Failure Detection and Recovery It’s not enough to have backup systems. It’s also crucial to detect failures quickly. Modern systems employ monitoring tools and rely on distributed coordination systems such as Zookeeper or etcd to identify faults in real-time: once detected, recovery mechanisms are triggered to restore the service.

zookeeper: is (was?) used for secrets management. it was also used as a basic health check, but has been since been replaced.

You can install Kafka from https://kafka.apache.org/quickstart. Because Druid and Kafka both use Apache Zookeeper, I opted to use the Zookeeper deployment that comes with Druid, so didn’t start it with Kafka. Once running, I created two topics for me to post the data into, and for Druid to ingest from:

Please note that we use Zookeeper as the Governance Center.

Next, review deployment complexity such as DB-less versus database-backed deployments. For example, Kong does require running Cassandra or Postgres. Apigee requires Cassandra, Zookeeper, and Postgres to run, while other solutions like Express Gateway and Tyk only require Redis. Apache APISIX uses etcd as its data store, it stores and manages routing-related and plugin-related configurations in etcd in the Data Plane.