sqledge
ctlstore
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sqledge
-
SQLedge: Replicate Postgres to SQLite on the Edge
It's nice to see another pg proxy using the pgx parser (their src[1]) - I built one using this lib too. However, this implementation is missing a lot of low level features to be considered close to compatible, including: multi-query transactions, auth, TLS, extended query mode, query cancellation.
[1]: https://github.com/zknill/sqledge/blob/main/pkg/pgwire/postg...
- SQLedge: Postgres Replication to SQLite
ctlstore
-
SQLedge: Replicate Postgres to SQLite on the Edge
We replicated our MySQL database to a SQLite edge at Segment in ctlstore: https://github.com/segmentio/ctlstore
We considered tailing binlogs directly but there's so much cruft and complexity involved trying to translate between types and such at that end, once you even just get passed properly parsing the binlogs and maintaining the replication connection. Then you have to deal with schema management across both systems too. Similar sets of problems using PostgreSQL as a source of truth.
In the end we decided just to wrap the whole thing up and abstract away the schema with a common set of types and a limited set of read APIs. Biggest missing piece I regret not getting in was support for secondary indexes.
-
Sharing an SQLite database across containers is surprisingly brilliant
> it is only practical for situations where the write rate (<100/s total) and data volumes (<10GB total) are low.
This comment from the GitHub project page is pretty important. Configuration data often sees slow change, and isn't huge so a custom approach seems viable. I wonder how close they are to that 100/s ceiling.
There's also an unmentioned transition to eventual consistency happening here:
> The implications of this decoupling is that the data at each instance is usually slightly out-of-date (by 1-2 seconds).
> The reader API provides a way to fetch an approximate staleness measurement that is accurate to within ~5 seconds.
That's could lead to more complex application logic or risk of confusing users with stale behavior. No free lunch here.
[1] https://segment.com/blog/separating-our-data-and-control-pla...
[2] https://github.com/segmentio/ctlstore
-
Go port of SQLite without CGo
at segment we benchmarked https://github.com/segmentio/ctlstore against this driver. We saw about a 50% hit to read performance, so we didn't move forward with it, but the improvements in service build times were really appealing.
What are some alternatives?
marmot - A distributed SQLite replicator built on top of NATS
go-sqlite - pure-Go SQLite driver for Go (SQLite embedded)
libpg_query - C library for accessing the PostgreSQL parser outside of the server environment
sqlite
awesome-db-tools - Everything that makes working with databases easier
sysroot - Files for cross-compilation
litefs - FUSE-based file system for replicating SQLite databases across a cluster of machines
libc