Inserting One Billion Rows in SQLite Under a Minute

• fast-sqlite3-inserts

  11 363 0.0 Rust

  Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way

• sqlitedatagen

  1 6 0.0 Go

  Super simple (~180 lines) Go util to generate SQLite data files filled with test data

  

You mention wanting to do a Go version. Not sure if it's useful, but this is a SQLite "bulk data generation" util I threw together ages ago in Go:

https://github.com/sqlitebrowser/sqlitedatagen

There's some initial work to parallelise it with goroutines here:

https://github.com/sqlitebrowser/sqlitedatagen/blob/multi_ro...

Didn't go very far down that track though, as the mostly single threaded nature of writes in SQLite seemed to prevent that from really achieving much. Well, I _think_ that's why it didn't really help. ;)

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• brimming

  1 5 5.4 Go

  Row Stuffing

Very nice! I shall give that a try. I always get excited by SQLite and go projects.

I maintain a similar go tool for work, which I use to stuff around 1TB into MariaDB a time:

https://github.com/rcbensley/brimming

• remixdb

  1 91 1.3 C

  RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.

• sqlite_micro_logger_arduino

  2 161 3.3 C

  Fast and Lean Sqlite database logger for Arduino UNO and above

Potentially interesting:

https://github.com/siara-cc/sqlite_micro_logger_arduino

https://github.com/siara-cc/sqlite_micro_logger_arduino/blob...

This is a heavily subsetted implementation of SQLite3 that can read/write databases (presumably on SD cards) from very small microcontrollers.

It presumably doesn't have the same ACID compliance properties, but with a single <1.5k source file, may represent a particularly efficient way to rapidly learn the intrinsics.

Now I'm thinking it could actually be interesting to see what drh thinks of this implementation (and any gotchas in it) because of its small size and accessibility.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project