s3fs
gcsfuse
| s3fs | gcsfuse | |
|---|---|---|
| 7 | 31 | |
| 814 | 1,981 | |
| 1.4% | 1.1% | |
| 7.8 | 9.7 | |
| 22 days ago | about 21 hours ago | |
| Python | Go | |
| BSD 3-clause "New" or "Revised" License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
s3fs
- Read files from s3 using Pandas/s3fs or AWS Data Wrangler?
- Gcsfuse: A user-space file system for interacting with Google Cloud Storage
-
what's the best python client for AWS automation these days?
- https://github.com/fsspec/s3fs (used by `pandas`, wraps aiobotocore)
- High-level, file-system like interface for S3 with AsyncIO support to replace/extend `boto3`
- Show HN: Query SQLite files stored in S3
-
Getting 403 return code from head_object even with s3:ListBucket permission
I'm using Python's s3fs library to check if a particular file exists in s3 with s3fs.S3FileSystem().exists(path), but I'm getting a Forbidden exception. From the stack trace, I can see it fails when calling s3's head_object method. The documentation for head_object method says:
gcsfuse
-
Gcsfuse: A user-space file system for interacting with Google Cloud Storage
It uses FUSE and there's three types of Kernel cache you could use with FUSE (although, it seems like gcsfuse is exposing only one):
1. Cache of file attributes in the Kernel (this is controlled by "stat-cache-ttl" value - https://github.com/GoogleCloudPlatform/gcsfuse/blob/7dc5c7ff...)
-
Does Cloud file store have life cycle management feature ?
You're looking at Filestore because your software can only write to a mounted file system? If so, you could mount a Google Cloud Storage bucket with Fuse. I haven't used Fuse myself in production, but it may be worth trying out for your workload.
-
Google Cloud Storage FUSE
Is this the same gcsfuse that's been around for years, only now with official Google support?
https://github.com/GoogleCloudPlatform/gcsfuse
- Suggestions on data transfer between VM instances
-
RANT: MICROSOFT'S INABILITY TO SUPPORT THEIR OWN HARDWARE IS GOING TO KILL ME
I'm pretty sure the storage for a stopped VM vs a disk image will be the same. Cheaper if you can store your data in GCS bucket? Take a look at GCS-Fuse to mount storage buckets into a VM.
-
Ongoing Incident in Google Cloud
Currently being tracked here: https://github.com/GoogleCloudPlatform/gcsfuse/issues/961
What are some alternatives?
goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go
google-drive-ocamlfuse - FUSE filesystem over Google Drive
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
s3www - Serve static files from any S3 compatible object storage services (Let's Encrypt ready)
juicefs - JuiceFS is a distributed POSIX file system built on top of Redis and S3.
aws-sdk-go-v2 - AWS SDK for the Go programming language.
afero - A FileSystem Abstraction System for Go
django-s3file - A lightweight file upload input for Django and Amazon S3
fsnotify - Cross-platform file system notifications for Go.
s3-proxy - S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)
go-systemd - Go bindings to systemd socket activation, journal, D-Bus, and unit files