SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Stream - Scalable APIs for Chat, Feeds, Moderation, & Video.
Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.
getstream.io
featured
InfluxDB – Built for High-Performance Time Series Workloads
InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
www.influxdata.com
featured
  1. LevelDB

    LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

    Google didnt, thus Chrome started replacing sqlite with https://github.com/google/leveldb

  2. Stream

    Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

    Stream logo
  3. fast-sqlite3-inserts

    Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way

    I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.

    I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/

    the repo link - https://github.com/avinassh/fast-sqlite3-inserts

  4. dqlite

    Embeddable, replicated and fault-tolerant SQL engine.

    I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite

  5. planetiler

    Flexible tool to build planet-scale vector tilesets from OpenStreetMap data fast

    I spent a while optimizing sqlite inserts for planetiler, this is what I came up with:

    https://github.com/onthegomap/planetiler/blob/db0ab02263baaa...

    It batches inserts into bulk statements and is able to do writes in the 500k+ per second range, and reads are 300-400k/s using those settings.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Marmot: Multi-writer distributed SQLite based on NATS

    4 projects | news.ycombinator.com | 11 Dec 2023
  • Strong Consistency with Raft and SQLite

    6 projects | news.ycombinator.com | 21 Mar 2023
  • 9 years of open-source database development: reviewing the designs

    2 projects | /r/programming | 10 Mar 2023
  • Cloud database for tomorrow's applications (written in Rust)

    7 projects | news.ycombinator.com | 22 Aug 2022
  • High-Availability SQLite

    1 project | news.ycombinator.com | 25 Apr 2022

Did you know that C++ is
the 7th most popular programming language
based on number of references?