fast-sqlite3-inserts
LevelDB
Our great sponsors
| fast-sqlite3-inserts | LevelDB | |
|---|---|---|
| 11 | 27 | |
| 363 | 35,046 | |
| - | 1.2% | |
| 0.0 | 0.0 | |
| about 1 year ago | 10 days ago | |
| Rust | C++ | |
| MIT License | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fast-sqlite3-inserts
-
SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.
I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/
the repo link - https://github.com/avinassh/fast-sqlite3-inserts
-
I/O is no longer the bottleneck
I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.
For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.
I published my findings here [1].
[0] - https://github.com/avinassh/fast-sqlite3-inserts
[1] - https://avi.im/blag/2021/fast-sqlite-inserts/
- Ask HN: Which personal projects got you hired?
-
Is there any language that is as similar as possible to Python in syntax, readability, and features, but is statically typed?
I have a side project where I tried to insert one billion rows in SQLite. I was able to insert 100 million rows using Python under 210 seconds. The same thing with PyPy took 120 seconds. I am wondering what kind of speed boost I would get with Cython
- Ask for benchmark. The owner can’t verify a 18% perf gain, could you?
-
Inserting One Billion Rows in SQLite Under A Minute
Measure, measure, measure! There is a PR which made really minor changes, but it got 2x speed boost with CPython version
- Inserting One Billion Rows in SQLite Under a Minute
- Weekly Coders, Hackers & All Tech related thread - 17/07/2021
-
How we achieved write speeds of 1.4 million rows per second
[somewhat related] Recently, I was benchmarking SQLite inserts and I managed to insert 3.3M records per second (100M in 33 ish seconds) on my local machine - https://github.com/avinassh/fast-sqlite3-inserts Ofcourse the comparison is not apples to apples, but sharing here if anyone finds it interesting
LevelDB
-
Codebases to read
I'm partial to how cleanly written https://github.com/google/leveldb is. It is a reasonable size to fully read & grok in not too long.
-
Unpacking LSM-Trees: The Powerhouse Behind Modern Databases
[4] leveldb/doc/impl.md at main · google/leveldb. GitHub. Retrieved October 21, 2023 from https://github.com/google/leveldb/blob/main/doc/impl.md
- Bloom filter support to leveldb by Sanjay Ghemawat
- SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
-
The Witty Guide to Installing LevelDB on Ubuntu: HostRooster® Edition
git clone https://github.com/google/leveldb.git
-
Is there a lightweight, stable and embedded database library?
leveldb?
- Ask HN: What's the best source code you've read?
-
LevelDB VS ZoneTree - a user suggested alternative
2 projects | 22 Aug 2022
- Is Mongo as popular in the job world as it is with tutorial makers?
-
Open Source Databases in Go
goleveldb - Implementation of the LevelDB key/value database in Go.
What are some alternatives?
tsbs - Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
RocksDB - A library that provides an embeddable, persistent key-value store for fast storage.
julia - The Julia Programming Language
MongoDB - The MongoDB Database
plum - Multiple dispatch in Python
Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
sqlite_micro_logger_arduino - Fast and Lean Sqlite database logger for Arduino UNO and above
SQLite - Unofficial git mirror of SQLite sources (see link for build instructions)
remixdb - RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.
LMDB - Read-only mirror of official repo on openldap.org. Issues and pull requests here are ignored. Use OpenLDAP ITS for issues.
dynamic-dns - An automated dynamic DNS solution for Docker and DigitalOcean
CouchDB - Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability