sqlite-s3-query
sqlite3vfshttp
Our great sponsors
sqlite-s3-query | sqlite3vfshttp | |
---|---|---|
5 | 5 | |
239 | 173 | |
- | - | |
7.5 | 1.8 | |
about 1 month ago | about 1 year ago | |
Python | Go | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sqlite-s3-query
-
Using Lidar to map tree shadows
Awesome project!
What about hosting the data in an S3 bucket with "Requestor Pays"? You'd only have the storage cost.
Disables anonymous access (so would a Dropbox share) but reduces your cost massively.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
You wouldn't necessarily need an SQL frontend as it's readonly anyway, and there are multiple ways of letting SQLite access databases in S3 buckets, e.g. https://github.com/michalc/sqlite-s3-query
-
Show HN: Query SQLite files stored in S3
Aha! This is done in https://github.com/uktrade/mbtiles-s3-server, but using an alternative library to query SQLite on S3, https://github.com/michalc/sqlite-s3-query
(Full disclosure: I wrote most of both of these)
-
Show HN: Mbtiles-S3-server – serve vector map tiles directly from mbtiles on S3
I was wondering how this ran queries against SQLite files in an S3 bucket. Here's the answer - it's using some very clever Python ctypes code to implement that HTTP Range header trick from last year:
https://github.com/michalc/sqlite-s3-query/blob/main/sqlite_...
Here's the inspiration for that: https://github.com/phiresky/sql.js-httpvfs
- Ws4sqlite: Query SQLite via HTTP
- Show HN: SQLite-S3-query – Python function to query a SQLite database on S3
sqlite3vfshttp
-
SQLite FTS5 Faster Than Redis Full-Text Search
I'm a big fan of FTS5 as a cheap/easy way of adding full text search to datasets.
Recently I've been storing these datasets in S3 and been querying them directly via a VFS[0]. Its a simple way to have FTS available to lambda functions without the cost of using an online database.
[0]: https://github.com/psanford/sqlite3vfshttp
- Deploying Strapi to AWS with AppPack
-
Ws4sqlite: Query SQLite via HTTP
You can also access sqlite databases directly from an http server that supports range requests (like s3). There are a bunch of implementations of this in different languages including Go[0] and Javascript[1].
[0]: https://github.com/psanford/sqlite3vfshttp
[1]: https://github.com/phiresky/sql.js-httpvfs
-
Ask HN: What could a modern database do that PostgreSQL and MySQL can't
There's a bunch of projects that have implemented this. I wrote a SQLite VFS in Go that lets you query a read-only SQLite db over http (including from s3) [0].
The VFS API offers the possibility for weirder storage solutions, if thats the type of thing you're into. Recently I've been moving some of my personal websites hosted on AWS Lambda over to use a read/write sqlite db backed by DynamoDB[1]. There are a bunch of limitations to this type of thing (like it uses a global write lock), but it works nicely for DBs that have low write frequency.
[0]: https://github.com/psanford/sqlite3vfshttp
[1]: https://github.com/psanford/donutdb
-
Show HN: SQLite-S3-query – Python function to query a SQLite database on S3
I just implemented something similar in Go[0]. Just yesterday I added the ability to compile it as a loadable sqlite3 extension, so you can use the normal sqlite3 cli tool to query databases hosted on an http(s) connection.
[0]: https://github.com/psanford/sqlite3vfshttp
What are some alternatives?
athena-sqlite - A SQLite driver for S3 and Amazon Athena 😳
Hasura - Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
tuql - Automatically create a GraphQL server from a SQLite database or a SQL file
sql.js-httpvfs - Hosting read-only SQLite databases on static file hosters like Github Pages
donutdb - Store and query a sqlite db directly backed by DynamoDB.
ws4sqlite - Query sqlite via json+http
ngx_http_mbtiles_module - A nginx module to serve map tiles directly from mbtiles container files
meteor-mysql - Reactive MySQL for Meteor
go-sqlite3 - sqlite3 driver for go using database/sql
absurd-sql - sqlite3 in ur indexeddb (hopefully a better backend soon)