maelstrom VS nosqlbench

Maelstrom [1], a workbench for learning distributed systems from the creator of Jepsen, includes a simple (model-checked) implementation of Raft and an excellent tutorial on implementing it.

Raft is a simple algorithm, but as others have noted, the original paper includes many correctness details often brushed over in toy implementations. Furthermore, the fallibility of real-world hardware (handling memory/disk corruption and grey failures), the requirements of real-world systems with tight latency SLAs, and a need for things like flexible quorum/dynamic cluster membership make implementing it for production a long and daunting task. The commit history of etcd and hashicorp/raft, likely the two most battle-tested open source implementations of raft that still surface correctness bugs on the regular tell you all you need to know.

The tigerbeetle team talks in detail about the real-world aspects of distributed systems on imperfect hardware/non-abstracted system models, and why they chose viewstamp replication, which predates Paxos but looks more like Raft.

[1]: https://github.com/jepsen-io/maelstrom/

[2]: https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/DE...

I've tried to run the echo example and I'me getting some problems. I assume it works as the same example in the maelstrom's getting started ready (https://github.com/jepsen-io/maelstrom/blob/main/doc/01-getting-ready/index.md).

Love it. Thanks for putting this together! The actual challenges here [0].

Though I'm curious: are these different from the chapters in the Maelstrom documentation [1]? There seems to be a bit of overlap anyway.

[0] https://fly.io/dist-sys/

[1] https://github.com/jepsen-io/maelstrom#documentation

Not directly related, but https://github.com/nosqlbench/nosqlbench is very flexible benchmark tool for Cassandra and other distributed systems

For performance testing, we also run common benchmarks: the popular YCSB (Yahoo! Cloud Serving Benchmark), NoSQLBench, LinkBench, SysBench, TPC-H, and TPC-C. We also run C Bench, our own Tarantool API benchmark. Its primitive operations are written in C, and scripts are described in Lua.

We used NoSQLBench to perform moderate load benchmarks. It comes with a convenient Docker image that we could use straight away to run stress jobs in our k8s cluster.

Jonathan Shook created NoSQLBench to be a cross-platform performance testing tool that is easier to use than cassandra-stress and (much) more powerful than YCSB; in fact, its scripting layer is powerful enough to support things that no other testing tool can enable, with particular emphasis on modeling complex workloads with fidelity, as well as simulating realistic scenarios such as load spikes. As its name suggests, NoSQLBench is not Cassandra-specific and encourages participation from all who want to contribute; today there are clients for Cassandra, CockroachDB, JDBC, and MongoDB, as well as non-database products Kafka and Pulsar. If you’re serious about performance testing in 2021, you should check out NoSQLBench. You can get started at GitHub. Other useful links: releases, discord, docs.