sqlite-s3-query
verneuil
| sqlite-s3-query | verneuil | |
|---|---|---|
| 5 | 5 | |
| 240 | 392 | |
| - | 1.0% | |
| 7.5 | 6.7 | |
| about 2 months ago | 2 months ago | |
| Python | C | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sqlite-s3-query
-
Using Lidar to map tree shadows
Awesome project!
What about hosting the data in an S3 bucket with "Requestor Pays"? You'd only have the storage cost.
Disables anonymous access (so would a Dropbox share) but reduces your cost massively.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
You wouldn't necessarily need an SQL frontend as it's readonly anyway, and there are multiple ways of letting SQLite access databases in S3 buckets, e.g. https://github.com/michalc/sqlite-s3-query
-
Show HN: Query SQLite files stored in S3
Aha! This is done in https://github.com/uktrade/mbtiles-s3-server, but using an alternative library to query SQLite on S3, https://github.com/michalc/sqlite-s3-query
(Full disclosure: I wrote most of both of these)
-
Show HN: Mbtiles-S3-server – serve vector map tiles directly from mbtiles on S3
I was wondering how this ran queries against SQLite files in an S3 bucket. Here's the answer - it's using some very clever Python ctypes code to implement that HTTP Range header trick from last year:
https://github.com/michalc/sqlite-s3-query/blob/main/sqlite_...
Here's the inspiration for that: https://github.com/phiresky/sql.js-httpvfs
- Ws4sqlite: Query SQLite via HTTP
- Show HN: SQLite-S3-query – Python function to query a SQLite database on S3
verneuil
- Show HN: Query SQLite files stored in S3
-
Embedded database with VFS support?
It'd be process wide. If you want an example can check out the example using a vfs here. There's an explicit passing of vfs there and an implicit usage of it. https://github.com/backtrace-labs/verneuil/blob/main/examples/rusqlite_integration.rs
- LiteFS a FUSE-based file system for replicating SQLite
-
A database for 2022 · Tailscale
It doesn't even have to be WAL-based system. Backtrace Labs has a SQLite virtual file system (VFS) called Verneuil that works similarly but works with the rollback journal instead of the WAL.
-
Ask HN: P2P Databases?
https://github.com/backtrace-labs/verneuil/ is one way to address the diffing / read replica part of the problem. I believe it's compatible with gossipping: most of the data is in small content-addressed chunks, with small manifests that tell clients what chunks to fetch and how to reassemble them to recreate a sqlite database. There's already client-side caching to persistent storage, and chunks can be fetched on demand.
Sharing replication data P2P, while retaining the simplicity of a single authoritative writer per database, is explicitly part of the project's long-term goals!
What are some alternatives?
sqlite3vfshttp - Go sqlite3 http vfs: query sqlite databases over http with range headers
litefs - FUSE-based file system for replicating SQLite databases across a cluster of machines
athena-sqlite - A SQLite driver for S3 and Amazon Athena 😳
go-ds-crdt - A distributed go-datastore implementation using Merkle-CRDTs.
tuql - Automatically create a GraphQL server from a SQLite database or a SQL file
dqlite - Embeddable, replicated and fault-tolerant SQL engine.
sql.js-httpvfs - Hosting read-only SQLite databases on static file hosters like Github Pages
WCDB - WCDB is a cross-platform database framework developed by WeChat.
ws4sqlite - Query sqlite via json+http
bb-remote-execution - Tools for Buildbarn to allow remote execution of build actions
ngx_http_mbtiles_module - A nginx module to serve map tiles directly from mbtiles container files
s3fs - S3 Filesystem