SSDB VS sqlite

I've used SSDB[0] in the past for some really stupid large datasets (20TB)_and it worked really well in production

[0] https://github.com/ideawu/ssdb

> Or wanting to shift the architecture entirely to avoid using memory-bound Redis as a queue with an overflow risk.

I wanted to use SSDB[1] instead of Redis for that reason, but it doesn't support the necessary data structures.

[1] https://github.com/ideawu/ssdb

There is also ssdb https://github.com/ideawu/ssdb

It's 2021 and we have extremely fast key-value lookups using LevelDB/RocksDB, but we're still using RAM-based caching tools [1] [2] [3]. It's time to consider RAM-based caching harmful, and start caching with SSDs for larger datasets and lower costs. For ex: SSDB [4]

[1] https://redis.io/

[2] https://memcached.org/

[3] https://news.ycombinator.com/item?id=29532552

[4] https://github.com/ideawu/ssdb#ssdb-vs-redis

Aside from this particular use-case, which is what most people here are talking about -- I had never heard of the SSDB project, but it interests me because I often use Redis for certain things it's able to do, aside from its in-memory storage, and sometimes the fact that the data is not stored on disk is a bit of a drawback for me, something I have to work around.

> You can get substantially better performance out of sqlite by using the lower level https://github.com/crawshaw/sqlite, turning on WAL etc, using a connection per goroutine for reads, and sending batches of writes over a buffered channel / queue to a dedicated writer thread. That way you can turn off SQLite’s built in per-connection mutex but still be thread safe since each connection is only used on a single thread at a time.

Would this protect against a row update in the middle of a read? e.g. would a row at least be internally consistent at the time it was read?

The default go sqlite driver is https://github.com/mattn/go-sqlite3, which is quite lovely, but I ran into issues with concurrency on read only databases.

I'm now using https://github.com/crawshaw/sqlite and it seems to address those issues (but I haven't gotten around to setting up a proper test to confirm). It may be worth perusing if you do run into performance problems. It does come with the caveat of not being a database/sql driver though.

crawshaw/sqlite - Low-level Go interface to SQLite

crawshaw/sqlite actually uses CGo -- it's a wrapper around the C version of SQLite. For example, see https://github.com/crawshaw/sqlite/blob/23d646f8ac00d9dd2390...

zombiezen/go-sqlite uses cznic's pure Go converted version of SQLite, so avoids CGo. It's explicitly stated to be "a fork of crawshaw.io/sqlite that uses modernc.org/sqlite, a CGo-free SQLite package. It aims to be a mostly drop-in replacement for crawshaw.io/sqlite."

The second well known driver is https://github.com/crawshaw/sqlite