s3fs
aws-sdk-go-v2
| s3fs | aws-sdk-go-v2 | |
|---|---|---|
| 7 | 14 | |
| 814 | 2,395 | |
| 1.4% | 1.7% | |
| 7.8 | 9.9 | |
| 20 days ago | 1 day ago | |
| Python | Go | |
| BSD 3-clause "New" or "Revised" License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
s3fs
- Read files from s3 using Pandas/s3fs or AWS Data Wrangler?
- Gcsfuse: A user-space file system for interacting with Google Cloud Storage
-
what's the best python client for AWS automation these days?
- https://github.com/fsspec/s3fs (used by `pandas`, wraps aiobotocore)
- High-level, file-system like interface for S3 with AsyncIO support to replace/extend `boto3`
- Show HN: Query SQLite files stored in S3
-
Getting 403 return code from head_object even with s3:ListBucket permission
I'm using Python's s3fs library to check if a particular file exists in s3 with s3fs.S3FileSystem().exists(path), but I'm getting a Forbidden exception. From the stack trace, I can see it fails when calling s3's head_object method. The documentation for head_object method says:
aws-sdk-go-v2
-
Updating your programs for S3 Express One Zone
It turns out that the AWS SDK for go, aws-sdk-go-v2, released some breaking changes that are unrelated to S3 Express One Zone. You can read more about these changes here.
-
AWS S3 (SDK v2) in Go Quickly example
AWS S3 official documentation AWS S3 official examples
-
AWS open source newsletter, #182
assume_role_with_mfa is a handy small MFA GUI tool from Cristian Magherusan-Stanciu designed to be used as credential_proces for AWS CLI configuration profiles. This is a workaround for the official AWS MFA setup, that works for the CLI but does not work for the Go SDK, as reported at aws-sdk-go-v2/#2356.
- AWS Golang 11/15/23 updates are incompatible
- Will error handling ever change/improve?
-
Use Golang for data processing with Amazon SNS and AWS Lambda
In this blog post, you will be using the aws-lambda-go library along with the AWS Go SDK v2 for an application that will process records from an Amazon SNS topic and store them in a DynamoDB table.
-
Use Golang for data processing with Amazon Kinesis and AWS Lambda
This blog post is for folks interested in learning how to use Golang and AWS Lambda to build a serverless solution. You will be using the aws-lambda-go library along with the AWS Go SDK v2 for an application that will process records from an Amazon Kinesis data stream and store them in a DynamoDB table. But that's not all! You will also use Go bindings for AWS CDK to implement "Infrastructure-as-code" for the entire solution and deploy it with the AWS CDK CLI.
-
How to work with multiple modules in a single repo?
The trouble with importing modules from within the same repository is that Go modules won't be able to work out where to download them from. The require directives I mentioned are used extensively by aws-sdk-go-v2 if you're interested in an example.
-
Interfacing w/ AWS Parameter Store via REST API
You may prefer https://github.com/aws/aws-sdk-go-v2 .
-
MicroShift
I ran into it by switching from aws-sdk-go to aws-sdk-go-v2 and the binary jumped from 27Mb to 66Mb [1]
Granted fully featured app will likely use all of the module's code so it's not a factor.
[1] https://github.com/aws/aws-sdk-go-v2/discussions/1532
What are some alternatives?
goofys - a high-performance, POSIX-ish Amazon S3 file system written in Go
minio-go - MinIO Go client SDK for S3 compatible object storage
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
kelseyhightower/envconfig - Golang library for managing configuration data from environment variables
s3www - Serve static files from any S3 compatible object storage services (Let's Encrypt ready)
s3fs - S3 FileSystem (fs.FS) implementation
django-s3file - A lightweight file upload input for Django and Amazon S3
s3fs - Amazon S3 filesystem for PyFilesystem2
s3-proxy - S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)
apt-transport-ipfs - IPFS transport for apt
verneuil - Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.
murder - Large scale server deploys using BitTorrent and the BitTornado library (NOTE: project no longer maintained)