async-std VS tigerbeetle

But maybe check out the discussion here https://github.com/async-rs/async-std/pull/631 or something (the blog post was linked on the end of it)

Have a look at these: https://github.com/async-rs/async-std/tree/main/examples

In fact, there are a lot of crates in Rust where in other programming languages, it would be included in the standard library. Examples are regex, random number generators, additional iterator methods, macros for other collections, num traits, loggers, HTTP libraries, error handling, async runtimes, serialization and deserialization, date and time, and many more.

Two major projects (non std lib but extremely commonly used) stand out in the area of async programming: Async std and Tokio - no doubt familiar to anyone that has turned an eye towards Rust for a second too long. Async architecture in general is likely very familiar to JavaScript programmers but in Rust there are some extra considerations (like ownership of the data that is thrown into an async function). Tokio is fast becoming a heavily supported and road tested async framework, with a thread scheduling runtime "baked in" that has learned from the history of Go, Erlang, and Java thread schedulers.

Ah... https://github.com/tokio-rs/tokio and https://github.com/async-rs/async-std ?

async-std: Basically a Tokio alternative with a few different design decisions.

Go's solution is for the scheduler to notice after a while when a goroutine has blocked execution and to shift goroutines waiting their turn to another thread. async-std pondered a similar approach with tasks, but it proved controversial and was never merged.

This indicates curl, zlib, openssl, and libnghttp2 as well as a bunch of WASM-related things are being dynamically linked into my executable. To resolve this, I looked at the build features exposed by surf and found that it selects the "curl_client" feature by default, which can be turned off and replaced with "h1-client-rustls" which uses an HTTP client backed by rustls and async-std and no dynamically linked libraries. Enabling this build feature removed all -sys dependencies from androidx-release-watcher, allowing me to build static executables of it.

And also, the actual PR never got merged.

Async in rust needs a runtime (aka executor) to run. You can maybe get a better description from the rust docs. As an example, Tokio attempts to provide an interface for a developer that is minimal change to the more common blocking code. So you'd end up putting #[tokio::main] above your main function to spin up the executor and most of the rest of the code is similar to a non-async version with a few sprinkles of .await, which you can see in the hello world for tokio. In contrast, async-std provides a more hands-on/low-level approach. If you are unlucky enough to have libraries that choose different stacks to work on, you'll possibly (probably?) have to handle both.

Indeed!

I was so glad to see you cite not only the Rebello paper but also Protocol-Aware Recovery for Consensus-Based Storage. When I read your first comment, I was about to reply to mention PAR, and then saw you had saved me the trouble!

UW-Madison are truly the vanguard where consensus hits the disk.

We implemented Protocol-Aware Recovery for TigerBeetle [1], and I did a talk recently at the Recurse Center diving into PAR, talking about the intersection of global consensus protocol and local storage engine. It's called Let's Remix Distributed Database Design! [2] and owes the big ideas to UW-Madison.

[1] https://github.com/coilhq/tigerbeetle

[2] https://www.youtube.com/watch?v=rNmZZLant9o

> It sounds like payments might be part of the larger concept of declarative programming (DP)

Yes, exactly! The idea with TigerBeetle's state machine [1] is to expose double-entry accounting as higher level financial primitives, so that developers can think in terms of declaring transfers from one account to another. The business logic behind the scenes is detailed, but the interfaces and data structures are simple.

[1] https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma...

> Maybe TigerBeetle could be generalized to support any multi-step distributed process?

That's part of the plan, that the distributed database framework of TigerBeetle can be used as a ”distributed Iron Man suit” to support any kind of state machine.

It's a pleasure. Let me know if you have any more questions about TigerBeetle. Our design doc is also here: https://github.com/coilhq/tigerbeetle/blob/main/docs/DESIGN....

Surprisingly, some of the most powerful distributed systems algorithms or tools are actually deterministic. They're powerful because they can "load the dice" and so make the distributed system more intuitive for humans to reason about, more resilient to real world network faults, and do all this with more performance.

For example, Barbara Liskov and James Cowling's deterministic view change [1], which isn't plagued by the latency issues of RAFT's randomized dueling leader problem. Viewstamped Replication Revisited's deterministic view change can react to a failed primary much quicker than RAFT (heartbeat timeouts don't require randomized "padding" as they do in RAFT), commence the leader election, and also ensure that the leader election succeeds without a split vote.

Determinism makes all that possible.

Deterministic testing [2][3] is also your best friend when it comes to testing distributed systems.

[1] I did a talk on VSR, including the benefits of the view change — https://www.youtube.com/watch?v=Wii1LX_ltIs

[2] FoundationDB are pioneers of deterministic testing — https://www.youtube.com/watch?v=OJb8A6h9jQQ

[3] TigerBeetle's deterministic simulation tests — https://github.com/coilhq/tigerbeetle#simulation-tests

This is the chasm problem, where people don't use a technology because people aren't using that technology, thus the technology has difficulty gaining adoption. I did see that Zig does have it's own killer app and startup that's using Zig: TigerBeattle.

Control flow statements should always be on their own lines, then it's easy to find all of them by visually scanning top-down, without needing to look all the way down each line.

[1]: https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/repl...

> In hindsight, data logic should be in the database itself.

This is the reason we are creating TigerBeetle [1] at Coil, as an open source distributed financial accounting database, with the double entry logic and financial invariants enforced through financial primitives within the database itself.

This is all the more critical for financial data, because raw data consistency is not enough for financial transactions, you also need financial consistency, not to mention immutability.

The performance of doing it this way is also easier. For example, around a million financial transactions per second on commodity hardware, with p100 latency around 10-20ms.

[1] https://github.com/coilhq/tigerbeetle

You probably already know this — because we’ve mentioned it a few times — but Coil champions and supports open-source projects and is privacy-first, by default. Over the years, Developer Relations at Coil has championed and sponsored teams that write Open Web Documentations and projects that empower open-source developers to get paid. Coil has also incubated many open-source projects like Tigerbeetle and Rafiki.

[6] Partial logical sector reads/writes even when using O_DIRECT — https://github.com/coilhq/tigerbeetle/blob/main/src/storage....