tigerbeetle VS too-many-lists

I'm waiting for someone to implement the Redis API by swapping out the state machine in TigerBeetle (which was built modularly such that the state machine can be swapped out).

https://tigerbeetle.com/

I fully agree with what Prime says at the end - Joran has really set a new bar here for all future database presentations.

Hearing that the entire TigerBeetle domain logic lives in a single file [0] (and is intended to be pluggable for other OLTP use cases!) makes it 1000% more tempting to spend the weekend getting up to speed with Zig.

[0] https://github.com/tigerbeetle/tigerbeetle/blob/main/src/sta...

Why would you want to build your own accounting ledger from scratch? Accounting is a completely new domain for most engineers, and TigerBeetle (https://tigerbeetle.com/) already solves this problem.

Maelstrom [1], a workbench for learning distributed systems from the creator of Jepsen, includes a simple (model-checked) implementation of Raft and an excellent tutorial on implementing it.

Raft is a simple algorithm, but as others have noted, the original paper includes many correctness details often brushed over in toy implementations. Furthermore, the fallibility of real-world hardware (handling memory/disk corruption and grey failures), the requirements of real-world systems with tight latency SLAs, and a need for things like flexible quorum/dynamic cluster membership make implementing it for production a long and daunting task. The commit history of etcd and hashicorp/raft, likely the two most battle-tested open source implementations of raft that still surface correctness bugs on the regular tell you all you need to know.

The tigerbeetle team talks in detail about the real-world aspects of distributed systems on imperfect hardware/non-abstracted system models, and why they chose viewstamp replication, which predates Paxos but looks more like Raft.

[1]: https://github.com/jepsen-io/maelstrom/

[2]: https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/DE...

> There is no reason to use a memory unsafe language anymore, except legacy codebases, and that is also slowly but surely diminishing. I'm still yet to hear this amazingly compelling reason that you just need memory unsafe languages. In terms of cost/benefits analysis, memory unsafety is literally all costs.

Tell that to the authors of new memory unsafe languages (like Zig) and creators of new project in those languages (like https://tigerbeetle.com) :(

You seem to have a preset opinion, and I'm not sure you are interested in re-evaluating it. So this is not written to change your mind.

I've developed production code in C, C++, Rust, and several other languages. And while like pretty much everything, there are situations where it's not a good fit, I find that the solutions tend to be the most robust and require the least post release debugging in Rust. That's my personal experience. It's not hard data. And yes occasionally it's annoying to please the compiler, and if there were no trait constraints or borrow rules, those instances would be easier. But way more often in my experience the compiler complained because my initial solution had problems I didn't realize before. So for me, these situations have been about going from building it the way I wanted to -> compiler tells me I didn't consider an edge case -> changing the implementation and or design to account for that edge case. Also using one example, where is Rust is notoriously hard and or un-ergonomic to use, and dismissing the entire language seems premature to me. For those that insist on learning Rust by implementing a linked list there is https://rust-unofficial.github.io/too-many-lists/.

Advent of Code was okay until I encounterd a problem that required a graph, tree or linked list to solve, where I hit a wall. Most coding exercises are similar--those requiring arrays and hashmaps and sets are okay, but complex data structures are a PITA. (There is an online course dedicated to linked lists in Rust but I couldn't grok it either). IMO you should simply skip problems that you can't solve with your current knowledge level and move on.

I feel obligated to point to the original cannon literature: https://rust-unofficial.github.io/too-many-lists/

I started learning Rust and like how the compiler is fussy about things. My plan was to implement the data structures I knew, but I got stuck at the singly linked list's remove() method. I've read the book as well, but I have no clue how to simplify this further:

My impression from the article is that Zig provides several different hashtables and not all of them are broken in this way.

This reminds me of Aria's comment in her Rust tutorial https://rust-unofficial.github.io/too-many-lists/ about failing to kill LinkedList. One philosophy (and the one Rust chose) for a stdlib is that this is only where things should live when they're so commonly needed that essentially everybody needs them either directly or to talk about. So, HashTable is needed by so much otherwise unrelated software that qualifies, BloomFilter, while it's real useful for some people, not so much. Aria cleaned out Rust's set of standard library containers before Rust 1.0, trying to keep only those most people would need. LinkedList isn't a good general purpose data structure, but, it was too popular and Aria was not able to remove it.

Having multiple hash tables feels like a win (they're optimized for different purposes) but may cost too much in terms of the necessary testing to ensure they all hit the quality you want.

> Cyclic references can be dealt with runtime safety checks too - like Rc and Weak.

Indeed. Starting out with code sprinkled with Rc, Weak, RefCell, etc is perfectly fine and performance will probably not be worse than in any other safe languages. And if you do this, Rust is pretty close to those languages in ease of use for what are otherwise complex topics in Rust.

A good reference for different approaches is Learn Rust With Entirely Too Many Linked Lists https://rust-unofficial.github.io/too-many-lists/

Second, once you've finished something introductory like The Book, read Learning Rust With Entirely Too Many Linked Lists. It really helped me to understand what ownership and borrowing actually mean in practical terms. If you don't mind paying for learning materials, a lot of people recommend Programming Rust, Second Edition by Blandy, Orendorff, and Tindall as either a complement, follow-up, or alternative to The Book.

Learning Rust With Entirely Too Many Linked Lists which highlights a lot of the differences with how you need to structure your code in Rust compared to other languages.