ctlstore VS awesome-db-tools

We replicated our MySQL database to a SQLite edge at Segment in ctlstore: https://github.com/segmentio/ctlstore

We considered tailing binlogs directly but there's so much cruft and complexity involved trying to translate between types and such at that end, once you even just get passed properly parsing the binlogs and maintaining the replication connection. Then you have to deal with schema management across both systems too. Similar sets of problems using PostgreSQL as a source of truth.

In the end we decided just to wrap the whole thing up and abstract away the schema with a common set of types and a limited set of read APIs. Biggest missing piece I regret not getting in was support for secondary indexes.

> it is only practical for situations where the write rate (<100/s total) and data volumes (<10GB total) are low.

This comment from the GitHub project page is pretty important. Configuration data often sees slow change, and isn't huge so a custom approach seems viable. I wonder how close they are to that 100/s ceiling.

There's also an unmentioned transition to eventual consistency happening here:

> The implications of this decoupling is that the data at each instance is usually slightly out-of-date (by 1-2 seconds).

> The reader API provides a way to fetch an approximate staleness measurement that is accurate to within ~5 seconds.

That's could lead to more complex application logic or risk of confusing users with stale behavior. No free lunch here.

[1] https://segment.com/blog/separating-our-data-and-control-pla...

[2] https://github.com/segmentio/ctlstore

at segment we benchmarked https://github.com/segmentio/ctlstore against this driver. We saw about a 50% hit to read performance, so we didn't move forward with it, but the improvements in service build times were really appealing.

#. SQLite WAL mode

From https://www.sqlite.org/isolation.html https://news.ycombinator.com/item?id=32247085 :

> [sqlite] WAL mode permits simultaneous readers and writers. It can do this because changes do not overwrite the original database file, but rather go into the separate write-ahead log file. That means that readers can continue to read the old, original, unaltered content from the original database file at the same time that the writer is appending to the write-ahead log

#. superfly/litefs: aFUSE-based file system for replicating SQLite https://github.com/superfly/litefs

#. sqldiff: https://www.sqlite.org/sqldiff.html https://news.ycombinator.com/item?id=31265005

#. dolthub/dolt: https://github.com/dolthub/dolt

> Dolt can be set up as a replica of your existing MySQL or MariaDB database using standard MySQL binlog replication. Every write becomes a Dolt commit. This is a great way to get the version control benefits of Dolt and keep an existing MySQL or MariaDB database.

#. pganalyze/libpg_query: https://github.com/pganalyze/libpg_query :

> C library for accessing the PostgreSQL parser outside of the server environment

#. Ibis + Substrait [ + DuckDB ]

> ibis strives to provide a consistent interface for interacting with a multitude of different analytical execution engines, most of which (but not all) speak some dialect of SQL.

> Today, Ibis accomplishes this with a lot of help from `sqlalchemy` and `sqlglot` to handle differences in dialect, or we interact directly with available Python bindings (for instance with the pandas, datafusion, and polars backends).

> [...] `Substrait` is a new cross-language serialization format for communicating (among other things) query plans. It's still in its early days, but there is already nascent support for Substrait in Apache Arrow, DuckDB, and Velox.

#. benbjohnson/postlite: https://github.com/benbjohnson/postlite

> postlite is a network proxy to allow access to remote SQLite databases over the Postgres wire protocol. This allows GUI tools to be used on remote SQLite databases which can make administration easier.

> The proxy works by translating Postgres frontend wire messages into SQLite transactions and converting results back into Postgres response wire messages. Many Postgres clients also inspect the pg_catalog to determine system information so Postlite mirrors this catalog by using an attached in-memory database with virtual tables. The proxy also performs minor rewriting on these system queries to convert them to usable SQLite syntax.

> Note: This software is in alpha. Please report bugs. Postlite doesn't alter your database unless you issue INSERT, UPDATE, DELETE commands so it's probably safe. If anything, the Postlite process may die but it shouldn't affect your database.

#. > "Hosting SQLite Databases on GitHub Pages" (2021) re: sql.js-httpvfs, DuckDB https://news.ycombinator.com/item?id=28021766

#. awesome-db-tools https://github.com/mgramin/awesome-db-tools

Check IDE and GUI sections of https://github.com/mgramin/awesome-db-tools, some of them should be able to work with SQLite. My use case is different from yours (I'm developer and have no need for manual data entry) so I'm not sure what's best for you, try them yourself.

Looking forward to my special db tools list https://github.com/mgramin/awesome-db-tools especially IDE and GUI sections

I have been starting to gathering learning resources and puzzles for SQL. Feel free to add new ones https://github.com/mgramin/awesome-db-tools#learning

I think sharing database access across Internet is quite dangerous decision because of security. Instead of LAMP (or something like that) you might to use special tools or platforms for building/generating API or even a whole application over your DB https://github.com/mgramin/awesome-db-tools (API and "Application platforms" sections)

I have an idea to create special section for SQL learning resources and SQL puzzles in my database tools list https://github.com/mgramin/awesome-db-tools/issues/336 because I see similar questions everyday. Lets do it together! 😉

Take a look at awesome-db-tools list, especially IDE section https://github.com/mgramin/awesome-db-tools

BTW I'm planning to add special section with SQL coding conventions and rules to my list https://github.com/mgramin/awesome-db-tools