fast-sqlite3-inserts
dqlite
Our great sponsors
fast-sqlite3-inserts | dqlite | |
---|---|---|
11 | 33 | |
363 | 3,716 | |
- | 1.5% | |
0.0 | 9.5 | |
about 1 year ago | 4 days ago | |
Rust | C | |
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fast-sqlite3-inserts
-
SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.
I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/
the repo link - https://github.com/avinassh/fast-sqlite3-inserts
-
I/O is no longer the bottleneck
I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.
For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.
I published my findings here [1].
[0] - https://github.com/avinassh/fast-sqlite3-inserts
[1] - https://avi.im/blag/2021/fast-sqlite-inserts/
- Ask HN: Which personal projects got you hired?
-
Is there any language that is as similar as possible to Python in syntax, readability, and features, but is statically typed?
I have a side project where I tried to insert one billion rows in SQLite. I was able to insert 100 million rows using Python under 210 seconds. The same thing with PyPy took 120 seconds. I am wondering what kind of speed boost I would get with Cython
- Ask for benchmark. The owner can’t verify a 18% perf gain, could you?
-
Inserting One Billion Rows in SQLite Under A Minute
Measure, measure, measure! There is a PR which made really minor changes, but it got 2x speed boost with CPython version
- Inserting One Billion Rows in SQLite Under a Minute
- Weekly Coders, Hackers & All Tech related thread - 17/07/2021
-
How we achieved write speeds of 1.4 million rows per second
[somewhat related] Recently, I was benchmarking SQLite inserts and I managed to insert 3.3M records per second (100M in 33 ish seconds) on my local machine - https://github.com/avinassh/fast-sqlite3-inserts Ofcourse the comparison is not apples to apples, but sharing here if anyone finds it interesting
dqlite
-
Marmot: Multi-writer distributed SQLite based on NATS
If you're interested in this, here are some related projects that all take slightly different approaches:
- LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io
- dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io
- cr-sqlite is a Rust-based loadable extension that adds CRDT changeset generation and reconciliation to SQLite: https://github.com/vlcn-io/cr-sqlite
Slightly related but not really (no multi writer, no C-level SQLite API or other restrictions):
- comdb2 (Bloombergs multi-homed RDMS using SQLite as the frontend)
- rqlite: RDMS with HTTP API and SQLite as the storage engine, used for replication and strong consistency (does not scale writes)
- litestream/LiteFS: disaster recovery replication
- liteserver: active read-only replication (predecessor of LiteSync)
- I'm All-In on Server-Side SQLite
-
SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite
- Strong Consistency with Raft and SQLite
-
9 years of open-source database development: reviewing the designs
Anyone knows how the DB this is about, https://rqlite.io/, compares with https://dqlite.io/ by Canonical (both seem to be distributed versions of sqlite)?
- SQLite the only database you will ever need in most cases
-
Transcending Posix: The End of an Era?
For folks' context, the new tool that's being discussed in the thread mentioned by the parent here is litefs [0], as well as which you can also look at rqlite [1] and dqlite [2], which all provide different trade-offs (e.g. rqlite is 'more strongly consistent' than litefs).
[0]: https://github.com/superfly/litefs
[1]: https://github.com/rqlite/rqlite
[2]: https://github.com/canonical/dqlite
-
SQLite is not a toy database
I presume you're familiar with https://github.com/canonical/dqlite (made by my employer) and https://github.com/rqlite/rqlite (unrelated)? How will mvsqlite compare to those?
-
GitDB, a distributed embeddable database on top of Git
Check out dqlite, it's sqlite but with a raft consensus to distribute changes through a log: https://dqlite.io/ You can link it in as a library too, it sounds like exactly what you want.
- Ask HN: Free and open source distributed database written in C++ or C
What are some alternatives?
tsbs - Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
rqlite - The lightweight, distributed relational database built on SQLite.
julia - The Julia Programming Language
kine - Run Kubernetes on MySQL, Postgres, sqlite, dqlite, not etcd.
plum - Multiple dispatch in Python
better-sqlite3 - The fastest and simplest library for SQLite3 in Node.js.
sqlite_micro_logger_arduino - Fast and Lean Sqlite database logger for Arduino UNO and above
litestream - Streaming replication for SQLite.
remixdb - RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.
boringproxy - Simple tunneling reverse proxy with a fast web UI and auto HTTPS. Designed for self-hosters.
dynamic-dns - An automated dynamic DNS solution for Docker and DigitalOcean
Bedrock - Rock solid distributed database specializing in active/active automatic failover and WAN replication