baseplate.go VS Apache ZooKeeper

We chose to use govips which is a cgo wrapper around the libvips image manipulation library. The majority of new development for services in our backend is written using baseplate.go. But Go is not an ideal choice for media processing as it cannot keep up with the performance of native code. The most widely used image-processing libraries like libmagick are primarily written in C or C++. Speed was a major factor in selecting libvips in order to keep latency low on CDN cache misses for images. In our tests, libvips was 3–4 times faster than libmagick on basic image processing operations. Content-aware smart cropping was implemented by porting smartcrop.js to Go. This is the only operation implemented in pure Go.

Even Reddit has python backends https://github.com/reddit/baseplate.py based on Pyramid. They also have a go one. https://github.com/reddit/baseplate.go

there's a multitude of services in reddit's architecture. as far as i can tell, they mostly using reddit's baseplate framework (which has implementations in both python and go).

FWIW, I believe Reddit has rewritten some Python into Go because of scaling issues. I don't know of any public write ups to confirm it, but at the least there's a public repo that contains their base template.

Gazette Inference Service is a baseplate.go (Reddit’s golang web services framework) thrift service whose single responsibility is serving ML inference requests to it’s clients. It is deployed with Reddit’s modern kubernetes infrastructure.

Baseplate is implemented in Python and Go, and although they share the same main functionality, smaller features differ between the two. One such feature that was previously on the Go implementation but not Python was deadline budget propagation, which passes on the remaining timeout available from the initial client request all the way through the server and any other requests that may follow. The lack of this feature in Baseplate.py meant that many resources were being wasted by servers doing unnecessary work, despite clients no longer awaiting their response due to timeout.

Apache ZooKeeper — a distributed coordination, synchronization, and configuration service (written in Java);

To use Kafka, we also need to deploy a service that keeps configuration informations such as Zookeeper.

Failure Detection and Recovery It’s not enough to have backup systems. It’s also crucial to detect failures quickly. Modern systems employ monitoring tools and rely on distributed coordination systems such as Zookeeper or etcd to identify faults in real-time: once detected, recovery mechanisms are triggered to restore the service.

zookeeper: is (was?) used for secrets management. it was also used as a basic health check, but has been since been replaced.

You can install Kafka from https://kafka.apache.org/quickstart. Because Druid and Kafka both use Apache Zookeeper, I opted to use the Zookeeper deployment that comes with Druid, so didn’t start it with Kafka. Once running, I created two topics for me to post the data into, and for Druid to ingest from:

Please note that we use Zookeeper as the Governance Center.

Next, review deployment complexity such as DB-less versus database-backed deployments. For example, Kong does require running Cassandra or Postgres. Apigee requires Cassandra, Zookeeper, and Postgres to run, while other solutions like Express Gateway and Tyk only require Redis. Apache APISIX uses etcd as its data store, it stores and manages routing-related and plugin-related configurations in etcd in the Data Plane.

ZooKeeper, a system for coordinating distributed nodes, similar to Google's Chubby

[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target

ElasticJob achieves distributed coordination through ZooKeeper. In practical scenarios, users may start multiple jobs in the same project simultaneously, all of which use the same Apache Curator client. There are certain risks due to the nature of ZooKeeper and the callback method of Curator in a single event thread.