SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s

1. LevelDB

   1 29 37,848 3.6 C++

   LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

   

   Google didnt, thus Chrome started replacing sqlite with https://github.com/google/leveldb

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. fast-sqlite3-inserts

   2 11 394 0.0 Rust

   Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way

   

   I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.

   I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/

   the repo link - https://github.com/avinassh/fast-sqlite3-inserts

4. dqlite

   3 33 4,089 8.9 C

   Embeddable, replicated and fault-tolerant SQL engine.

   

   I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite

5. planetiler

   4 34 1,650 9.4 Java

   Flexible tool to build planet-scale vector tilesets from OpenStreetMap data fast

   

   I spent a while optimizing sqlite inserts for planetiler, this is what I came up with:

   https://github.com/onthegomap/planetiler/blob/db0ab02263baaa...

   It batches inserts into bulk statements and is able to do writes in the 500k+ per second range, and reads are 300-400k/s using those settings.

Suggest a related project