SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s

• LevelDB

  27 35,186 0.0 C++

  LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

  

Google didnt, thus Chrome started replacing sqlite with https://github.com/google/leveldb

• fast-sqlite3-inserts

  11 363 0.0 Rust

  Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way

I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.

I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/

the repo link - https://github.com/avinassh/fast-sqlite3-inserts

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• dqlite

  33 3,721 9.5 C

  Embeddable, replicated and fault-tolerant SQL engine.

  

I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite

• planetiler

  30 1,156 9.3 Java

  Flexible tool to build planet-scale vector tilesets from OpenStreetMap data fast

  

I spent a while optimizing sqlite inserts for planetiler, this is what I came up with:

https://github.com/onthegomap/planetiler/blob/db0ab02263baaa...

It batches inserts into bulk statements and is able to do writes in the 500k+ per second range, and reads are 300-400k/s using those settings.

Suggest a related project