s3fs
sqlite-s3-query
| s3fs | sqlite-s3-query | |
|---|---|---|
| 7 | 5 | |
| 814 | 240 | |
| 1.4% | - | |
| 7.8 | 7.5 | |
| 22 days ago | about 2 months ago | |
| Python | Python | |
| BSD 3-clause "New" or "Revised" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
s3fs
- Read files from s3 using Pandas/s3fs or AWS Data Wrangler?
- Gcsfuse: A user-space file system for interacting with Google Cloud Storage
-
what's the best python client for AWS automation these days?
- https://github.com/fsspec/s3fs (used by `pandas`, wraps aiobotocore)
- High-level, file-system like interface for S3 with AsyncIO support to replace/extend `boto3`
- Show HN: Query SQLite files stored in S3
-
Getting 403 return code from head_object even with s3:ListBucket permission
I'm using Python's s3fs library to check if a particular file exists in s3 with s3fs.S3FileSystem().exists(path), but I'm getting a Forbidden exception. From the stack trace, I can see it fails when calling s3's head_object method. The documentation for head_object method says:
sqlite-s3-query
-
Using Lidar to map tree shadows
Awesome project!
What about hosting the data in an S3 bucket with "Requestor Pays"? You'd only have the storage cost.
Disables anonymous access (so would a Dropbox share) but reduces your cost massively.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
You wouldn't necessarily need an SQL frontend as it's readonly anyway, and there are multiple ways of letting SQLite access databases in S3 buckets, e.g. https://github.com/michalc/sqlite-s3-query
-
Show HN: Query SQLite files stored in S3
Aha! This is done in https://github.com/uktrade/mbtiles-s3-server, but using an alternative library to query SQLite on S3, https://github.com/michalc/sqlite-s3-query
(Full disclosure: I wrote most of both of these)
-
Show HN: Mbtiles-S3-server – serve vector map tiles directly from mbtiles on S3
I was wondering how this ran queries against SQLite files in an S3 bucket. Here's the answer - it's using some very clever Python ctypes code to implement that HTTP Range header trick from last year:
https://github.com/michalc/sqlite-s3-query/blob/main/sqlite_...
Here's the inspiration for that: https://github.com/phiresky/sql.js-httpvfs
- Ws4sqlite: Query SQLite via HTTP
- Show HN: SQLite-S3-query – Python function to query a SQLite database on S3
What are some alternatives?
goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go
sqlite3vfshttp - Go sqlite3 http vfs: query sqlite databases over http with range headers
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
athena-sqlite - A SQLite driver for S3 and Amazon Athena 😳
s3www - Serve static files from any S3 compatible object storage services (Let's Encrypt ready)
tuql - Automatically create a GraphQL server from a SQLite database or a SQL file
aws-sdk-go-v2 - AWS SDK for the Go programming language.
sql.js-httpvfs - Hosting read-only SQLite databases on static file hosters like Github Pages
django-s3file - A lightweight file upload input for Django and Amazon S3
ws4sqlite - Query sqlite via json+http
s3-proxy - S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)
ngx_http_mbtiles_module - A nginx module to serve map tiles directly from mbtiles container files