Our great sponsors
-
rclone
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
-
Seaweed File System
Discontinued SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. [Moved to: https://github.com/seaweedfs/seaweedfs] (by chrislusf)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
"I am an AWS administrator/architect for work so we are always trying to weigh pros and cons."
Very informal survey ... but I wonder if you are familiar with the 'rclone' tool:
https://rclone.org/
... just curious ...
There are simply too many object store vendors, with different storage pricing, egress pricing, network speed, and reliabilities.
What about using SeaweedFS gateway to remote object store? https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot...
Basically you can think it is a cache. You operate with normal S3 APIs, and the data will be uploaded to remote storage asynchronously (most vendors have free uploading).
It's not a magic pill for all users, but it should be very fast, and cut down the API costs, egress costs.