s3fs
mountpoint-s3
| s3fs | mountpoint-s3 | |
|---|---|---|
| 7 | 17 | |
| 814 | 4,017 | |
| 1.4% | 1.8% | |
| 7.8 | 9.5 | |
| 22 days ago | 4 days ago | |
| Python | Rust | |
| BSD 3-clause "New" or "Revised" License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
s3fs
- Read files from s3 using Pandas/s3fs or AWS Data Wrangler?
- Gcsfuse: A user-space file system for interacting with Google Cloud Storage
-
what's the best python client for AWS automation these days?
- https://github.com/fsspec/s3fs (used by `pandas`, wraps aiobotocore)
- High-level, file-system like interface for S3 with AsyncIO support to replace/extend `boto3`
- Show HN: Query SQLite files stored in S3
-
Getting 403 return code from head_object even with s3:ListBucket permission
I'm using Python's s3fs library to check if a particular file exists in s3 with s3fs.S3FileSystem().exists(path), but I'm getting a Forbidden exception. From the stack trace, I can see it fails when calling s3's head_object method. The documentation for head_object method says:
mountpoint-s3
-
Row Zero and Viewport Data Streaming
... or does "S3 file system" mean https://github.com/awslabs/mountpoint-s3 - a Rust project by AWS Labs that provides "a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system" ?
- s3m: A CLI for streams of data in S3 buckets
-
S3 Express Is All You Need
Looks like support for S3 Express was merged in with version 1.30 just a few hours ago https://github.com/awslabs/mountpoint-s3/pull/642
-
Gcsfuse: A user-space file system for interacting with Google Cloud Storage
mountpoint-s3 is AWS’ first party solution for mounting s3 buckets as file systems: https://github.com/awslabs/mountpoint-s3
Haven’t used it but it looks cool, if a bit immature.
- Mountpoint for S3
- When would something like this come to ADLS Gen 2?
-
Running Amazon S3 Mountpoint Inside a Container
FROM rust:1.68.0 as Build RUN apt-get update && apt-get install -y \ clang\ cmake \ curl \ fuse \ git \ libfuse-dev \ pkg-config \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* \ && git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git \ && cd mountpoint-s3 \ && cargo build --release FROM debian:bullseye-slim RUN apt-get update && apt-get install -y \ ca-certificates \ libfuse-dev \ sudo \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* COPY --from=build /mountpoint-s3/target/release/mount-s3 /usr/local/bin/mount-s3 RUN chmod 777 /usr/local/bin/mount-s3 RUN useradd -ms /bin/bash mount-s3-user \ && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \ && adduser mount-s3-user sudo USER mount-s3-user
- GitHub - awslabs/mountpoint-s3: A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.
-
The inside story on Mountpoint for Amazon S3, a high-performance open source file client
This might be useful with a MinIO server, although not directly supported
What are some alternatives?
goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go
s3fs-fuse - FUSE-based file system backed by Amazon S3
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
PosixSyncFS - PosixSyncFS is a set of Bash scripts that allow users to create a real POSIX filesystem and sync it to a remote storage bucket for backup and recovery purposes.
s3www - Serve static files from any S3 compatible object storage services (Let's Encrypt ready)
aws-sdk-go-v2 - AWS SDK for the Go programming language.
aws-eks-iam-auth-controller - Kubernetes operator which consolidates custom resources into `aws-auth` ConfigMap.
django-s3file - A lightweight file upload input for Django and Amazon S3
usbd - User-Space Block Device (USBD) Framework (written in Go)
s3-proxy - S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)