dqlite VS kine

If you're interested in this, here are some related projects that all take slightly different approaches:

- LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io

- dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io

- cr-sqlite is a Rust-based loadable extension that adds CRDT changeset generation and reconciliation to SQLite: https://github.com/vlcn-io/cr-sqlite

Slightly related but not really (no multi writer, no C-level SQLite API or other restrictions):

- comdb2 (Bloombergs multi-homed RDMS using SQLite as the frontend)

- rqlite: RDMS with HTTP API and SQLite as the storage engine, used for replication and strong consistency (does not scale writes)

- litestream/LiteFS: disaster recovery replication

- liteserver: active read-only replication (predecessor of LiteSync)

I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite

Anyone knows how the DB this is about, https://rqlite.io/, compares with https://dqlite.io/ by Canonical (both seem to be distributed versions of sqlite)?

For folks' context, the new tool that's being discussed in the thread mentioned by the parent here is litefs [0], as well as which you can also look at rqlite [1] and dqlite [2], which all provide different trade-offs (e.g. rqlite is 'more strongly consistent' than litefs).

[0]: https://github.com/superfly/litefs

[1]: https://github.com/rqlite/rqlite

[2]: https://github.com/canonical/dqlite

I presume you're familiar with https://github.com/canonical/dqlite (made by my employer) and https://github.com/rqlite/rqlite (unrelated)? How will mvsqlite compare to those?

Check out dqlite, it's sqlite but with a raft consensus to distribute changes through a log: https://dqlite.io/ You can link it in as a library too, it sounds like exactly what you want.

-- Kine (Kine is not etcd)

Looking at [1], is there any reason why this couldn’t run in something like DynamoDB? I feel like the operations required would fit

https://github.com/k3s-io/kine/blob/master/pkg/drivers/gener...

I don't like some of the decisions that k3s makes which is why I stay away from it, but it's a great project. I'm not using kine yet, but I fantasize about using it exclusively every time I deal with etcd-induced slowness/random CPU spikes on my group of controller nodes. k0s supports kine too of course but I just haven't switched it on yet.

Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0].

> I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer).

One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO.

If you do build a plugin system though, you can pass off the work of maintaining the other implementations!

[0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all

[1]: https://github.com/k3s-io/kine

I could spent >0s looking into this, but I've been curious how kline implements Etcd's Watch API. For the matter, I don't have much idea in general how kline handles data. Kline would be a great candidate for a blog post or introductory technical article on.

I do quickly see one issue- evidently watches presently use polling. The issue requests use of postgres triggers: https://github.com/k3s-io/kine/issues/20

check out https://github.com/k3s-io/kine

Not to be mistaken for high-availability Dqlite[1], which is one of the options one can run the k3s kubernetes distribution on (instead of etcd), via the Kine etcd shim[2]. Ultimately though the K3s team replaced Dqlite with an embedded etcd to get high-availability[3].

[1] https://dqlite.io/

[2] https://github.com/k3s-io/kine

[3] https://rancher.com/docs/k3s/latest/en/installation/ha-embed...